| Server IP : 198.38.94.13 / Your IP : 216.73.217.33 Web Server : Apache System : Linux d4744.dxb1.stableserver.net 5.14.0-611.49.1.el9_7.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Apr 21 16:39:08 EDT 2026 x86_64 User : revivere ( 1140) PHP Version : 8.2.33 Disable Function : NONE MySQL : OFF | cURL : ON | WGET : ON | Perl : ON | Python : OFF | Sudo : OFF | Pkexec : OFF Directory : /opt/saltstack/salt/lib/python3.10/site-packages/charset_normalizer/__pycache__/ |
Upload File : |
o
;jiR � @ sL d dl Z d dlmZ d dlmZmZmZmZmZ ddl m
Z
m Z m Z m
Z
ddlmZmZmZmZ ddlmZ ddlmZmZ dd lmZmZmZm Z m Z m Z m Z e � d
�Z!e �"� Z#e#�$e �%d ��
d(dee&e'f de(de(de)deee* deee* de+de+de)de+d efd d �Z,
d(d ede(de(de)deee* deee* de+de+de)de+d efd d!�Z-
d(d"ee*e&ef de(de(de)deee* deee* de+de+de)de+d efd#d$�Z.
d)d%eee*ee&f de(de(de)deee* deee* de+de+de)de+d e+fd&d'�Z/dS )*� N)�PathLike)�BinaryIO�List�Optional�Set�Union� )�coherence_ratio�encoding_languages�mb_encoding_languages�merge_coherence_ratios)�IANA_SUPPORTED�TOO_BIG_SEQUENCE�TOO_SMALL_SEQUENCE�TRACE)�
mess_ratio)� CharsetMatch�CharsetMatches)�any_specified_encoding�cut_sequence_chunks� iana_name�identify_sig_or_bom�
is_cp_similar�is_multi_byte_encoding�should_strip_sig_or_bomZcharset_normalizerz)%(asctime)s | %(levelname)s | %(message)s� � 皙�����?TF皙�����?� sequences�steps�
chunk_size� threshold� cp_isolation� cp_exclusion�preemptive_behaviour�explain�language_threshold�enable_fallback�returnc
/ C s� t | ttf�std�t| ����|r tj}
t�t � t�
t � t | �} | dkrGt�
d� |r;t�t � t�
|
p9tj� tt| dddg d�g�S |dur]t�t d d
�|�� d d � |D �}ng }|durut�t d
d
�|�� dd � |D �}ng }| || kr�t�t d||| � d}| }|dkr�| | |k r�t| | �}t | �tk } t | �tk}
| r�t�t d�| �� n |
r�t�t d�| �� g }|r�t| �nd}|dur�|�|� t�t d|� t� }g }g }d}d}d}t� }t| �\}}|du�r|�|� t�t dt |�|� |�d� d|v�r|�d� |t D �]�}|�r!||v�r!�q|�r+||v �r+�q||v �r2�q|� |� d}||k}|�oCt |�} |dv �rU|�sUt�t d|� �q|dv �rf|�sft�t d|� �qzt |�} W n t t!f�y� t�t d|� Y �qw z9|
�r�| du �r�t"| du �r�| dtd�� n | t |�td�� |d � nt"| du �r�| n| t |�d� |d �}W n+ t#t$f�y� } z t | t$��s�t�t d |t"| �� |�|� W Y d} ~ �qd} ~ ww d} |D ]
} t%|| ��r�d } n�q�| �rt�t d || � �qt&|�s dnt |�| t| | ��}!| �o#|du�o#t |�| k }"|"�r.t�t d |� tt |!�d! �}#t'|#d"�}#d}$d}%g }&g }'zLt(| ||!||| || |� D ]=}(|&�|(� |'�t)|(||d u �oodt |� k�omd"kn �� |'d# |k�r}|$d7 }$|$|#k�s�|�r�| du �r� n�qPW n! t#�y� } zt�t d$|t"| �� |#}$d }%W Y d} ~ nd} ~ ww |%�s�|
�r�| �s�z| td%�d� j*|d&d'� W n# t#�y� } zt�t d(|t"| �� |�|� W Y d} ~ �qd} ~ ww |'�r�t+|'�t |'� nd})|)|k�s|$|#k�rC|�|� t�t d)||$t,|)d* d+d,�� | �rA|dd|fv �rA|%�sAt| ||dg |�}*||k�r7|*}n
|dk�r?|*}n|*}�qt�t d-|t,|)d* d+d,�� | �sYt-|�}+nt.|�}+|+�rlt�t d.�|t"|+��� g },|dk�r�|&D ]}(t/|(||+�r�d/�|+�nd�}-|,�|-� �qut0|,�}.|.�r�t�t d0�|.|�� |�t| ||)||.|�� ||ddfv �r�|)d1k �r�t�
d2|� |�r�t�t � t�
|
� t|| g� S ||k�r�t�
d3|� |�r�t�t � t�
|
� t|| g� S �qt |�dk�rM|�s|�s|�r t�t d4� |�rt�
d5|j1� |�|� n2|�r#|du �s5|�r0|�r0|j2|j2k�s5|du�r@t�
d6� |�|� n
|�rMt�
d7� |�|� |�r_t�
d8|�3� j1t |�d � nt�
d9� |�rqt�t � t�
|
� |S ):af
Given a raw bytes sequence, return the best possibles charset usable to render str objects.
If there is no results, it is a strong indicator that the source is binary/not text.
By default, the process will extract 5 blocks of 512o each to assess the mess and coherence of a given sequence.
And will give up a particular code page after 20% of measured mess. Those criteria are customizable at will.
The preemptive behavior DOES NOT replace the traditional detection workflow, it prioritize a particular code page
but never take it for granted. Can improve the performance.
You may want to focus your attention to some code page or/and not others, use cp_isolation and cp_exclusion for that
purpose.
This function will strip the SIG in the payload/sequence every time except on UTF-16, UTF-32.
By default the library does not setup any handler other than the NullHandler, if you choose to set the 'explain'
toggle to True it will alter the logger configuration to add a StreamHandler that is suitable for debugging.
Custom logging format and handler can be set manually.
z4Expected object of type bytes or bytearray, got: {0}r z<Encoding detection on empty bytes, assuming utf_8 intention.�utf_8g F� Nz`cp_isolation is set. use this flag for debugging purpose. limited list of encoding allowed : %s.z, c S � g | ]}t |d ��qS �F�r ��.0Zcp� r1 �J/opt/saltstack/salt/lib/python3.10/site-packages/charset_normalizer/api.py�
<listcomp>[ � z from_bytes.<locals>.<listcomp>zacp_exclusion is set. use this flag for debugging purpose. limited list of encoding excluded : %s.c S r, r- r. r/ r1 r1 r2 r3 f r4 z^override steps (%i) and chunk_size (%i) as content does not fit (%i byte(s) given) parameters.r z>Trying to detect encoding from a tiny portion of ({}) byte(s).zIUsing lazy str decoding because the payload is quite large, ({}) byte(s).z@Detected declarative mark in sequence. Priority +1 given for %s.zIDetected a SIG or BOM mark on first %i byte(s). Priority +1 given for %s.�ascii> �utf_32�utf_16z\Encoding %s won't be tested as-is because it require a BOM. Will try some sub-encoder LE/BE.> �utf_7zREncoding %s won't be tested as-is because detection is unreliable without BOM/SIG.z2Encoding %s does not provide an IncrementalDecoderg �� A)�encodingz9Code page %s does not fit given bytes sequence at ALL. %sTzW%s is deemed too similar to code page %s and was consider unsuited already. Continuing!zpCode page %s is a multi byte encoding table and it appear that at least one character was encoded using n-bytes.� � ���zaLazyStr Loading: After MD chunk decode, code page %s does not fit given bytes sequence at ALL. %sg j�@�strict)�errorsz^LazyStr Loading: After final lookup, code page %s does not fit given bytes sequence at ALL. %szc%s was excluded because of initial chaos probing. Gave up %i time(s). Computed mean chaos is %f %%.�d � )�ndigitsz=%s passed initial chaos probing. Mean measured chaos is %f %%z&{} should target any language(s) of {}�,z We detected language {} using {}r z.Encoding detection: %s is most likely the one.zoEncoding detection: %s is most likely the one as we detected a BOM or SIG within the beginning of the sequence.zONothing got out of the detection process. Using ASCII/UTF-8/Specified fallback.z7Encoding detection: %s will be used as a fallback matchz:Encoding detection: utf_8 will be used as a fallback matchz:Encoding detection: ascii will be used as a fallback matchz]Encoding detection: Found %s as plausible (best-candidate) for content. With %i alternatives.z=Encoding detection: Unable to determine any suitable charset.)4�
isinstance� bytearray�bytes� TypeError�format�type�logger�level�
addHandler�explain_handler�setLevelr �len�debug�
removeHandler�logging�WARNINGr r �log�join�intr r r �append�setr r
�addr r �ModuleNotFoundError� ImportError�str�UnicodeDecodeError� LookupErrorr �range�maxr r �decode�sum�roundr
r r r r9 Z fingerprintZbest)/r r r! r" r# r$ r% r& r' r( Zprevious_logger_level�lengthZis_too_small_sequenceZis_too_large_sequenceZprioritized_encodingsZspecified_encodingZtestedZtested_but_hard_failureZtested_but_soft_failureZfallback_asciiZ fallback_u8Zfallback_specified�resultsZ sig_encodingZ sig_payloadZ
encoding_ianaZdecoded_payloadZbom_or_sig_availableZstrip_sig_or_bomZis_multi_byte_decoder�eZsimilar_soft_failure_testZencoding_soft_failedZr_Zmulti_byte_bonusZmax_chunk_gave_upZearly_stop_countZlazy_str_hard_failureZ md_chunksZ md_ratios�chunkZmean_mess_ratioZfallback_entryZtarget_languagesZ cd_ratiosZchunk_languagesZcd_ratios_mergedr1 r1 r2 �
from_bytes! s� ��
�� � �����
�
�
��� ��
��
��� �
�� ��
��
��
�
&��
�
� ������� �
��
�
�� �
�
��
�����
�
�
� �� �
��
�
rg �fpc
C s t | �� ||||||||| �
S )z�
Same thing than the function from_bytes but using a file pointer that is already ready.
Will not close the file pointer.
)rg �read)
rh r r! r" r# r$ r% r& r' r( r1 r1 r2 �from_fp� s �rj �pathc
C sH t | d��}
t|
||||||||| �
W d � S 1 s w Y dS )z�
Same thing than the function from_bytes but with one extra step. Opening and reading given file path in binary mode.
Can raise IOError.
�rbN)�openrj ) rk r r! r" r# r$ r% r& r' r( rh r1 r1 r2 � from_path s �$�rn �fp_or_path_or_payloadc
C s� t | ttf�rt| ||||||||| d�
}
|
S t | ttf�r0t| ||||||||| d�
}
|
S t| ||||||||| d�
}
|
S )a)
Detect if the given input (file, bytes, or path) points to a binary file. aka. not a string.
Based on the same main heuristic algorithms and default kwargs at the sole exception that fallbacks match
are disabled to be stricter around ASCII-compatible but unlikely to be a string.
) r r! r" r# r$ r% r& r' r( )rC r[ r rn rE rD rg rj ) ro r r! r" r# r$ r% r& r' r( Zguessesr1 r1 r2 � is_binary3 s\ �-������
rp ) r r r NNTFr T) r r r NNTFr F)0rQ �osr �typingr r r r r Zcdr r
r r Zconstantr
r r r Zmdr Zmodelsr r �utilsr r r r r r r � getLoggerrI �
StreamHandlerrL � setFormatter� FormatterrE rD rU �floatr[ �boolrg rj rn rp r1 r1 r1 r2 �<module> s $
��
����
�
��� �
�
� Z�����
�
��� �
�
� � ����
�
��� �
�
�!�����
�
��� �
� �