Wala shell

Advanced PHP File Manager

Server IP: 127.0.0.1 | PHP Version: 8.3.30 | Domains Hosted: 54

Name	Size	Last Modified	Actions
📁 ..	-	2024-10-09 06:13
📄 advanced_search.rst Advanced Search =============== Charset Normalizer method ``from_bytes``, ``from_fp`` and ``from_path`` provide some optional parameters that can be tweaked. As follow :: from charset_normalizer import from_bytes my_byte_str = 'Bсеки човек има право на образование.'.encode('cp1251') results = from_bytes( my_byte_str, steps=10, # Number of steps/block to extract from my_byte_str chunk_size=512, # Set block size of each extraction threshold=0.2, # Maximum amount of chaos allowed on first pass cp_isolation=None, # Finite list of encoding to use when searching for a match cp_exclusion=None, # Finite list of encoding to avoid when searching for a match preemptive_behaviour=True, # Determine if we should look into my_byte_str (ASCII-Mode) for pre-defined encoding explain=False, # Print on screen what is happening when searching for a match language_threshold=0.1 # Minimum coherence ratio / language ratio match accepted ) Using CharsetMatches ------------------------------ Here, ``results`` is a ``CharsetMatches`` object. It behave like a list but does not implements all related methods. Initially, it is sorted. Calling ``best()`` is sufficient to extract the most probable result. .. autoclass:: charset_normalizer.CharsetMatches :members: List behaviour -------------- Like said earlier, ``CharsetMatches`` object behave like a list. :: # Call len on results also work if not results: print('No match for your sequence') # Iterate over results like a list for match in results: print(match.encoding, 'can decode properly your sequence using', match.alphabets, 'and language', match.language) # Using index to access results if results: print(str(results[0])) Using best() ------------ Like said above, ``CharsetMatches`` object behave like a list and it is sorted by default after getting results from ``from_bytes``, ``from_fp`` or ``from_path``. Using ``best()`` return the most probable result, the first entry of the list. Eg. idx 0. It return a ``CharsetMatch`` object as return value or None if there is not results inside it. :: result = results.best() Calling first() --------------- The very same thing than calling the method ``best()``. Class aliases ------------- ``CharsetMatches`` is also known as ``CharsetDetector``, ``CharsetDoctor`` and ``CharsetNormalizerMatches``. It is useful if you prefer short class name. Verbose output -------------- You may want to understand why a specific encoding was not picked by charset_normalizer. All you have to do is passing ``explain`` to True when using methods ``from_bytes``, ``from_fp`` or ``from_path``.	2.72 KB	2024-10-09 06:13
📄 cli.rst Command Line Interface ====================== charset-normalizer ship with a CLI that should be available as `normalizer`. This is a great tool to fully exploit the detector capabilities without having to write Python code. Possible use cases: #. Quickly discover probable originating charset from a file. #. I want to quickly convert a non Unicode file to Unicode. #. Debug the charset-detector. Down below, we will guide you through some basic examples. Arguments --------- You may simply invoke `normalizer -h` (with the h(elp) flag) to understand the basics. :: usage: normalizer [-h] [-v] [-a] [-n] [-m] [-r] [-f] [-t THRESHOLD] file [file ...] The Real First Universal Charset Detector. Discover originating encoding used on text file. Normalize text to unicode. positional arguments: files File(s) to be analysed optional arguments: -h, --help show this help message and exit -v, --verbose Display complementary information about file if any. Stdout will contain logs about the detection process. -a, --with-alternative Output complementary possibilities if any. Top-level JSON WILL be a list. -n, --normalize Permit to normalize input file. If not set, program does not write anything. -m, --minimal Only output the charset detected to STDOUT. Disabling JSON output. -r, --replace Replace file when trying to normalize it instead of creating a new one. -f, --force Replace file without asking if you are sure, use this flag with caution. -t THRESHOLD, --threshold THRESHOLD Define a custom maximum amount of chaos allowed in decoded content. 0. <= chaos <= 1. --version Show version information and exit. .. code:: bash normalizer ./data/sample.1.fr.srt You may also run the command line interface using: .. code:: bash python -m charset_normalizer ./data/sample.1.fr.srt Main JSON Output ---------------- 🎉 Since version 1.4.0 the CLI produce easily usable stdout result in JSON format. .. code:: json { "path": "/home/default/projects/charset_normalizer/data/sample.1.fr.srt", "encoding": "cp1252", "encoding_aliases": [ "1252", "windows_1252" ], "alternative_encodings": [ "cp1254", "cp1256", "cp1258", "iso8859_14", "iso8859_15", "iso8859_16", "iso8859_3", "iso8859_9", "latin_1", "mbcs" ], "language": "French", "alphabets": [ "Basic Latin", "Latin-1 Supplement" ], "has_sig_or_bom": false, "chaos": 0.149, "coherence": 97.152, "unicode_path": null, "is_preferred": true } I recommend the `jq` command line tool to easily parse and exploit specific data from the produced JSON. Multiple File Input ------------------- It is possible to give multiple files to the CLI. It will produce a list instead of an object at the top level. When using the `-m` (minimal output) it will rather print one result (encoding) per line. Unicode Conversion ------------------ If you desire to convert any file to Unicode you will need to append the flag `-n`. It will produce another file, it won't replace it by default. The newly created file path will be declared in `unicode_path` (JSON output).	3.61 KB	2024-10-09 06:13
📄 getstarted.rst Installation ============ This installs a package that can be used from Python (``import charset_normalizer``). To install for all users on the system, administrator rights (root) may be required. Using PIP --------- Charset Normalizer can be installed from pip:: pip install charset-normalizer You may retrieve the latest unicodedata backport as follow:: pip install charset-normalizer[unicode_backport] From git via master ----------------------- You can install from dev-master branch using git:: git clone https://github.com/Ousret/charset_normalizer.git cd charset_normalizer/ python setup.py install Basic Usage =========== The new way ----------- You may want to get right to it. :: from charset_normalizer import from_bytes, from_path # This is going to print out your sequence once properly decoded print( str( from_bytes( my_byte_str ).best() ) ) # You could also want the same from a file print( str( from_path( './data/sample.1.ar.srt' ).best() ) ) Backward compatibility ---------------------- If you were used to python chardet, we are providing the very same ``detect()`` method as chardet. This function is mostly backward-compatible with Chardet. The migration should be painless. :: from charset_normalizer import detect # This will behave exactly the same as python chardet result = detect(my_byte_str) if result['encoding'] is not None: print('got', result['encoding'], 'as detected encoding') You may upgrade your code with ease. CTRL + R ``from chardet import detect`` to ``from charset_normalizer import detect``.	1.7 KB	2024-10-09 06:13
📄 handling_result.rst ================ Handling Result ================ When initiating search upon a buffer, bytes or file you can assign the return value and fully exploit it. :: my_byte_str = 'Bсеки човек има право на образование.'.encode('cp1251') # Assign return value so we can fully exploit result result = from_bytes( my_byte_str ).best() print(result.encoding) # cp1251 Using CharsetMatch ---------------------------- Here, ``result`` is a ``CharsetMatch`` object or ``None``. .. autoclass:: charset_normalizer.CharsetMatch :members:	595 B	2024-10-09 06:13
📄 miscellaneous.rst ============== Miscellaneous ============== Convert to str -------------- Any ``CharsetMatch`` object can be transformed to exploitable ``str`` variable. :: my_byte_str = 'Bсеки човек има право на образование.'.encode('cp1251') # Assign return value so we can fully exploit result result = from_bytes( my_byte_str ).best() # This should print 'Bсеки човек има право на образование.' print(str(result)) Logging ------- Prior to the version 2.0.11 you may encounter some unexpected logs in your streams. Something along the line of: :: ... \| WARNING \| override steps (5) and chunk_size (512) as content does not fit (465 byte(s) given) parameters. ... \| INFO \| ascii passed initial chaos probing. Mean measured chaos is 0.000000 % ... \| INFO \| ascii should target any language(s) of ['Latin Based'] It is most likely because you altered the root getLogger instance. The package has its own logic behind logging and why it is useful. See https://docs.python.org/3/howto/logging.html to learn the basics. If you are looking to silence and/or reduce drastically the amount of logs, please upgrade to the latest version available for `charset-normalizer` using your package manager or by `pip install charset-normalizer -U`. The latest version will no longer produce any entry greater than `DEBUG`. On `DEBUG` only one entry will be observed and that is about the detection result. Then regarding the others log entries, they will be pushed as `Level 5`. Commonly known as TRACE level, but we do not register it globally. Detect binaries --------------- This package offers a neat way to detect files that can be considered as 'binaries' meaning that it is not likely to be a text-file. :: from charset_normalizer import is_binary # It can receive both a path or bytes or even a file pointer. result = is_binary("./my-file.ext") # This should print 'True' or 'False' print(result)	1.98 KB	2024-10-09 06:13
📄 support.rst ================= Support ================= If you are running: - Python >=2.7,<3.5: Unsupported - Python 3.5: charset-normalizer < 2.1 - Python 3.6: charset-normalizer < 3.1 Upgrade your Python interpreter as soon as possible. ------------------- Supported Encodings ------------------- Here are a list of supported encoding and supported language with latest update. Also this list may change depending of your python version. Charset Normalizer is able to detect any of those encoding. This list is NOT static and depends heavily on what your current cPython version is shipped with. See https://docs.python.org/3/library/codecs.html#standard-encodings =============== =============================================================================================================================== IANA Code Page Aliases =============== =============================================================================================================================== ascii 646, ansi_x3.4_1968, ansi_x3_4_1968, ansi_x3.4_1986, cp367, csascii, ibm367, iso646_us, iso_646.irv_1991, iso_ir_6, us, us_ascii big5 big5_tw, csbig5, x_mac_trad_chinese big5hkscs big5_hkscs, hkscs cp037 037, csibm037, ebcdic_cp_ca, ebcdic_cp_nl, ebcdic_cp_us, ebcdic_cp_wt, ibm037, ibm039 cp1026 1026, csibm1026, ibm1026 cp1125 1125, ibm1125, cp866u, ruscii cp1140 1140, ibm1140 cp1250 1250, windows_1250 cp1251 1251, windows_1251 cp1252 1252, windows_1252 cp1253 1253, windows_1253 cp1254 1254, windows_1254 cp1255 1255, windows_1255 cp1256 1256, windows_1256 cp1257 1257, windows_1257 cp1258 1258, windows_1258 cp273 273, ibm273, csibm273 cp424 424, csibm424, ebcdic_cp_he, ibm424 cp437 437, cspc8codepage437, ibm437 cp500 500, csibm500, ebcdic_cp_be, ebcdic_cp_ch, ibm500 cp775 775, cspc775baltic, ibm775 cp850 850, cspc850multilingual, ibm850 cp852 852, cspcp852, ibm852 cp855 855, csibm855, ibm855 cp857 857, csibm857, ibm857 cp858 858, csibm858, ibm858 cp860 860, csibm860, ibm860 cp861 861, cp_is, csibm861, ibm861 cp862 862, cspc862latinhebrew, ibm862 cp863 863, csibm863, ibm863 cp864 864, csibm864, ibm864 cp865 865, csibm865, ibm865 cp866 866, csibm866, ibm866 cp869 869, cp_gr, csibm869, ibm869 cp932 932, ms932, mskanji, ms_kanji cp949 949, ms949, uhc cp950 950, ms950 euc_jis_2004 jisx0213, eucjis2004, euc_jis2004 euc_jisx0213 eucjisx0213 euc_jp eucjp, ujis, u_jis euc_kr euckr, korean, ksc5601, ks_c_5601, ks_c_5601_1987, ksx1001, ks_x_1001, x_mac_korean gb18030 gb18030_2000 gb2312 chinese, csiso58gb231280, euc_cn, euccn, eucgb2312_cn, gb2312_1980, gb2312_80, iso_ir_58, x_mac_simp_chinese gbk 936, cp936, ms936 hp_roman8 roman8, r8, csHPRoman8 hz hzgb, hz_gb, hz_gb_2312 iso2022_jp csiso2022jp, iso2022jp, iso_2022_jp iso2022_jp_1 iso2022jp_1, iso_2022_jp_1 iso2022_jp_2 iso2022jp_2, iso_2022_jp_2 iso2022_jp_3 iso2022jp_3, iso_2022_jp_3 iso2022_jp_ext iso2022jp_ext, iso_2022_jp_ext iso2022_kr csiso2022kr, iso2022kr, iso_2022_kr iso8859_10 csisolatin6, iso_8859_10, iso_8859_10_1992, iso_ir_157, l6, latin6 iso8859_11 thai, iso_8859_11, iso_8859_11_2001 iso8859_13 iso_8859_13, l7, latin7 iso8859_14 iso_8859_14, iso_8859_14_1998, iso_celtic, iso_ir_199, l8, latin8 iso8859_15 iso_8859_15, l9, latin9 iso8859_16 iso_8859_16, iso_8859_16_2001, iso_ir_226, l10, latin10 iso8859_2 csisolatin2, iso_8859_2, iso_8859_2_1987, iso_ir_101, l2, latin2 iso8859_3 csisolatin3, iso_8859_3, iso_8859_3_1988, iso_ir_109, l3, latin3 iso8859_4 csisolatin4, iso_8859_4, iso_8859_4_1988, iso_ir_110, l4, latin4 iso8859_5 csisolatincyrillic, cyrillic, iso_8859_5, iso_8859_5_1988, iso_ir_144 iso8859_6 arabic, asmo_708, csisolatinarabic, ecma_114, iso_8859_6, iso_8859_6_1987, iso_ir_127 iso8859_7 csisolatingreek, ecma_118, elot_928, greek, greek8, iso_8859_7, iso_8859_7_1987, iso_ir_126 iso8859_8 csisolatinhebrew, hebrew, iso_8859_8, iso_8859_8_1988, iso_ir_138 iso8859_9 csisolatin5, iso_8859_9, iso_8859_9_1989, iso_ir_148, l5, latin5 iso2022_jp_2004 iso_2022_jp_2004, iso2022jp_2004 johab cp1361, ms1361 koi8_r cskoi8r kz1048 kz_1048, rk1048, strk1048_2002 latin_1 8859, cp819, csisolatin1, ibm819, iso8859, iso8859_1, iso_8859_1, iso_8859_1_1987, iso_ir_100, l1, latin, latin1 mac_cyrillic maccyrillic mac_greek macgreek mac_iceland maciceland mac_latin2 maccentraleurope, maclatin2 mac_roman macintosh, macroman mac_turkish macturkish ptcp154 csptcp154, pt154, cp154, cyrillic_asian shift_jis csshiftjis, shiftjis, sjis, s_jis, x_mac_japanese shift_jis_2004 shiftjis2004, sjis_2004, s_jis_2004 shift_jisx0213 shiftjisx0213, sjisx0213, s_jisx0213 tis_620 tis620, tis_620_0, tis_620_2529_0, tis_620_2529_1, iso_ir_166 utf_16 u16, utf16 utf_16_be unicodebigunmarked, utf_16be utf_16_le unicodelittleunmarked, utf_16le utf_32 u32, utf32 utf_32_be utf_32be utf_32_le utf_32le utf_8 u8, utf, utf8, utf8_ucs2, utf8_ucs4 (+utf_8_sig) utf_7* u7, unicode-1-1-utf-7 cp720 N.A. cp737 N.A. cp856 N.A. cp874 N.A. cp875 N.A. cp1006 N.A. koi8_r N.A. koi8_t N.A. koi8_u N.A. =============== =============================================================================================================================== *: Only if a SIG/mark is found. ------------------- Supported Languages ------------------- Those language can be detected inside your content. All of these are specified in ./charset_normalizer/assets/__init__.py . \| English, \| German, \| French, \| Dutch, \| Italian, \| Polish, \| Spanish, \| Russian, \| Japanese, \| Portuguese, \| Swedish, \| Chinese, \| Ukrainian, \| Norwegian, \| Finnish, \| Vietnamese, \| Czech, \| Hungarian, \| Korean, \| Indonesian, \| Turkish, \| Romanian, \| Farsi, \| Arabic, \| Danish, \| Serbian, \| Lithuanian, \| Slovene, \| Slovak, \| Malay, \| Hebrew, \| Bulgarian, \| Croatian, \| Hindi, \| Estonian, \| Thai, \| Greek, \| Tamil. ---------------------------- Incomplete Sequence / Stream ---------------------------- It is not (yet) officially supported. If you feed an incomplete byte sequence (eg. truncated multi-byte sequence) the detector will most likely fail to return a proper result. If you are purposely feeding part of your payload for performance concerns, you may stop doing it as this package is fairly optimized. We are working on a dedicated way to handle streams.	6.89 KB	2024-10-09 06:13

Terminal