U hh@shdZddlZddlmZddlmZmZmZm Z m Z GdddeZ d e e efe ee dd d ZdS) zDocument loader helpers.N)Path)List NamedTupleOptionalUnioncastc@s2eZdZUdZeeed<eed<eeed<dS) FileEncodingz File encoding as the NamedTuple.encoding confidencelanguageN)__name__ __module__ __qualname____doc__rstr__annotations__floatrrP/tmp/pip-unpacked-wheel-9gdii04g/langchain_community/document_loaders/helpers.pyrs   r) file_pathtimeoutreturnc sddlt|}tttdfdd }tjJ}|||}z|j|d}Wn&tjj k rvt d|YnXW5QRXt dd |Drt d |d d |DS) aTry to detect the file encoding. Returns a list of `FileEncoding` tuples with the detected encodings ordered by confidence. Args: file_path: The path to the file to detect the encoding for. timeout: The timeout in seconds for the encoding detection. rN)rrc s2t|d}|}W5QRXttt|S)Nrb)openreadrrdictZ detect_all)rfrawdatachardetrrread_and_detect#s z.detect_file_encodings..read_and_detect)rz-Timeout reached while detecting encoding for css|]}|ddkVqdSr Nr).0r rrr 1sz(detect_file_encodings..zCould not detect encoding for cSs"g|]}|ddk rtf|qSr")r)r#encrrr 3s z)detect_file_encodings..) r rrr concurrentZfuturesZThreadPoolExecutorZsubmitresult TimeoutErrorall RuntimeError)rrr!executorfuture encodingsrrrdetect_file_encodingss   r/)r)rconcurrent.futuresr'pathlibrtypingrrrrrrrintr/rrrrs