U hQ@sddlZddlZddlZddlmZmZddlmZmZm Z m Z m Z ddl m Z ddlmZGdddeeZGdd d eZGd d d eZdS) N)ABCabstractmethod)DictIteratorOptionalTupleUnion)Document) BaseLoaderc@seZdZdZdddddddd d dddd d d eeeeeefeeeeeeeeefeeefeeefeeeed d d dZ e e dddZ e edddZeedddZeee e dddZd eeeee e dddZeeeefdddZd S)!DedocBaseLoadera Base Loader that uses `dedoc` (https://dedoc.readthedocs.io). Loader enables extracting text, tables and attached files from the given file: * `Text` can be split by pages, `dedoc` tree nodes, textual lines (according to the `split` parameter). * `Attached files` (when with_attachments=True) are split according to the `split` parameter. For attachments, langchain Document object has an additional metadata field `type`="attachment". * `Tables` (when with_tables=True) are not split - each table corresponds to one langchain Document object. For tables, Document object has additional metadata fields `type`="table" and `text_as_html` with table HTML representation. documentTF auto_tabbyrus+eng:autoN)split with_tableswith_attachmentsrecursion_deep_attachmentspdf_with_text_layerlanguagepagesis_one_column_documentdocument_orientationneed_header_footer_analysisneed_binarizationneed_pdf_table_analysis delimiterencoding) file_pathrrrrrrrrrrrrrrreturncCsddtD|_ddddh|_||jkrFtd|d|jd ||_||_||_|jdkrfd nd }||jd <||jd <dS)a Initialize with file path and parsing parameters. Args: file_path: path to the file for processing split: type of document splitting into parts (each part is returned separately), default value "document" "document": document text is returned as a single langchain Document object (don't split) "page": split document text into pages (works for PDF, DJVU, PPTX, PPT, ODP) "node": split document text into tree nodes (title nodes, list item nodes, raw text nodes) "line": split document text into lines with_tables: add tables to the result - each table is returned as a single langchain Document object Parameters used for document parsing via `dedoc` (https://dedoc.readthedocs.io/en/latest/parameters/parameters.html): with_attachments: enable attached files extraction recursion_deep_attachments: recursion level for attached files extraction, works only when with_attachments==True pdf_with_text_layer: type of handler for parsing PDF documents, available options ["true", "false", "tabby", "auto", "auto_tabby" (default)] language: language of the document for PDF without a textual layer and images, available options ["eng", "rus", "rus+eng" (default)], the list of languages can be extended, please see https://dedoc.readthedocs.io/en/latest/tutorials/add_new_language.html pages: page slice to define the reading range for parsing PDF documents is_one_column_document: detect number of columns for PDF without a textual layer and images, available options ["true", "false", "auto" (default)] document_orientation: fix document orientation (90, 180, 270 degrees) for PDF without a textual layer and images, available options ["auto" (default), "no_change"] need_header_footer_analysis: remove headers and footers from the output result for parsing PDF and images need_binarization: clean pages background (binarize) for PDF without a textual layer and images need_pdf_table_analysis: parse tables for PDF without a textual layer and images delimiter: column separator for CSV, TSV files encoding: encoding of TXT, CSV, TSV cSsi|]\}}|dkr||qS)>rrr self).0keyvaluer#r#N/tmp/pip-unpacked-wheel-9gdii04g/langchain_community/document_loaders/dedoc.py dsz,DedocBaseLoader.__init__..r pagenodelineGot $ for `split`, but should be one of ``treeZlinearstructure_typeZneed_content_analysisN)localsitemsparsing_parametersvalid_split_values ValueErrorrrr )r"r rrrrrrrrrrrrrrr0r#r#r'__init__#sA  zDedocBaseLoader.__init__r!c csddl}zddlm}Wntk r4tdYnX||d}d|jd_| }|j|j |j d|id }W5QRX|j | |jd EdHdS) Lazily load documents.rN) DedocManagerzE`dedoc` package not found, please install it with `pip install dedoc`)Zmanager_configTloggerZattachments_dir)r parameters document_treer)tempfileZdedocr9 ImportError _make_configconfigdisabledTemporaryDirectoryparser r3_split_documentZ to_api_schemadictr)r"r>r9Z dedoc_managerZtmpdirr=r#r#r' lazy_loadws$     zDedocBaseLoader.lazy_loadcCsdS)zu Make configuration for DedocManager according to the file extension and parsing parameters. Nr#r"r#r#r'r@szDedocBaseLoader._make_config) paragraphr!cs>dfdd|dD}|r2|dd|n|d}|S)z1Get text (recursively) of the document tree node. csg|]}|qSr#) _json2txt)r$ subparagraphrHr#r' sz-DedocBaseLoader._json2txt.. subparagraphstext)join)r"rIZsubparagraphs_textrOr#rHr'rKs zDedocBaseLoader._json2txt)r=document_metadatar!ccsRt|ddkr4|dD]}|j||dEdHqnt|d||ddVdS)z4Parse recursively document tree obtained by `dedoc`.rNrr=rQNrOmetadataZ page_contentrS)len_parse_subparagraphsr )r"r=rQrLr#r#r'rVs  z$DedocBaseLoader._parse_subparagraphs)r=radditional_metadatar!ccs|d}|r||}|dkrB|j|ddd}t||dVn|dkr|ddd}|d dd }d }|D]P} | dd |kr||| 7}qrt||d |idV| dd }|| }qrt||d |idVn|d kr|dddD]$} | d} t|| || dVqn@|d krF|j|dd|dEdHntd|d|jd|jr|ddD]0} || \} } t| | dd| ddVqr|dD]"}|j||jddidEdHqdS)z=Split document into parts according to the `split` parameter.rSr contentZ structure)rIrTr)rNrpage_idr+r*rRNr,r-r.Ztablestable)typeZ text_as_html attachmentsr\ attachment)r=rrW) rKr rVr5r4r _get_tablerEr)r"r=rrWrQrOZnodesrYZ page_textr*Z line_metadatar[ table_text table_htmlr^r#r#r'rEsl         zDedocBaseLoader._split_document)r[r!c Csd}|dD]:}|D](}|ddd|dD7}|d7}q|d7}q d }|dD]|}|d 7}|D]b}dd d|dD}t|}|d 7}|d r|d7}|d|dd|dd|d7}qd|d7}qT|d7}||fS)z.Get text and HTML representation of the table.rZcells css|]}|dVqdSrONr#r$r+r#r#r' sz-DedocBaseLoader._get_table..lines rJzK z css|]}|dVqdSrdr#rer#r#r'rfszz z z
)rPhtmlescape)r"r[r`rowcellraZ cell_textr#r#r'r_s,      zDedocBaseLoader._get_table)N)__name__ __module__ __qualname____doc__strboolrintrr6rr rGrrFr@rKrVrErr_r#r#r#r'r sb     T  Kr c@seZdZdZedddZdS)DedocFileLoaderaw DedocFileLoader document loader integration to load files using `dedoc`. The file loader automatically detects the file type (with the correct extension). The list of supported file types is gives at https://dedoc.readthedocs.io/en/latest/index.html#id1. Please see the documentation of DedocBaseLoader to get more details. Setup: Install ``dedoc`` package. .. code-block:: bash pip install -U dedoc Instantiate: .. code-block:: python from langchain_community.document_loaders import DedocFileLoader loader = DedocFileLoader( file_path="example.pdf", # split=..., # with_tables=..., # pdf_with_text_layer=..., # pages=..., # ... ) Load: .. code-block:: python docs = loader.load() print(docs[0].page_content[:100]) print(docs[0].metadata) .. code-block:: python Some text { 'file_name': 'example.pdf', 'file_type': 'application/pdf', # ... } Lazy load: .. code-block:: python docs = [] docs_lazy = loader.lazy_load() for doc in docs_lazy: docs.append(doc) print(docs[0].page_content[:100]) print(docs[0].metadata) .. code-block:: python Some text { 'file_name': 'example.pdf', 'file_type': 'application/pdf', # ... } r7cCs ddlm}||j|j|jdS)Nr)make_manager_config)r Zparsing_paramsr)Zdedoc.utils.langchainrur r3r)r"rur#r#r'r@`s  zDedocFileLoader._make_configN)rmrnrorprFr@r#r#r#r'rtsBrtcseZdZdZdddddddd d d dddd d d eeeeeeefeeeeeeeeefeeefeeefeeeed d fddZ e e dddZ e dddZeee eeeee effdddZZS)DedocAPIFileLoaderaU Load files using `dedoc` API. The file loader automatically detects the file type (even with the wrong extension). By default, the loader makes a call to the locally hosted `dedoc` API. More information about `dedoc` API can be found in `dedoc` documentation: https://dedoc.readthedocs.io/en/latest/dedoc_api_usage/api.html Please see the documentation of DedocBaseLoader to get more details. Setup: You don't need to install `dedoc` library for using this loader. Instead, the `dedoc` API needs to be run. You may use Docker container for this purpose. Please see `dedoc` documentation for more details: https://dedoc.readthedocs.io/en/latest/getting_started/installation.html#install-and-run-dedoc-using-docker .. code-block:: bash docker pull dedocproject/dedoc docker run -p 1231:1231 Instantiate: .. code-block:: python from langchain_community.document_loaders import DedocAPIFileLoader loader = DedocAPIFileLoader( file_path="example.pdf", # url=..., # split=..., # with_tables=..., # pdf_with_text_layer=..., # pages=..., # ... ) Load: .. code-block:: python docs = loader.load() print(docs[0].page_content[:100]) print(docs[0].metadata) .. code-block:: python Some text { 'file_name': 'example.pdf', 'file_type': 'application/pdf', # ... } Lazy load: .. code-block:: python docs = [] docs_lazy = loader.lazy_load() for doc in docs_lazy: docs.append(doc) print(docs[0].page_content[:100]) print(docs[0].metadata) .. code-block:: python Some text { 'file_name': 'example.pdf', 'file_type': 'application/pdf', # ... } zhttp://0.0.0.0:1231r TFr rrrrN)urlrrrrrrrrrrrrrr)r rwrrrrrrrrrrrrrrr!cs>tj|||||||| | | | | |||d||_d|jd<dS)a Initialize with file path, API url and parsing parameters. Args: file_path: path to the file for processing url: URL to call `dedoc` API split: type of document splitting into parts (each part is returned separately), default value "document" "document": document is returned as a single langchain Document object (don't split) "page": split document into pages (works for PDF, DJVU, PPTX, PPT, ODP) "node": split document into tree nodes (title nodes, list item nodes, raw text nodes) "line": split document into lines with_tables: add tables to the result - each table is returned as a single langchain Document object Parameters used for document parsing via `dedoc` (https://dedoc.readthedocs.io/en/latest/parameters/parameters.html): with_attachments: enable attached files extraction recursion_deep_attachments: recursion level for attached files extraction, works only when with_attachments==True pdf_with_text_layer: type of handler for parsing PDF documents, available options ["true", "false", "tabby", "auto", "auto_tabby" (default)] language: language of the document for PDF without a textual layer and images, available options ["eng", "rus", "rus+eng" (default)], the list of languages can be extended, please see https://dedoc.readthedocs.io/en/latest/tutorials/add_new_language.html pages: page slice to define the reading range for parsing PDF documents is_one_column_document: detect number of columns for PDF without a textual layer and images, available options ["true", "false", "auto" (default)] document_orientation: fix document orientation (90, 180, 270 degrees) for PDF without a textual layer and images, available options ["auto" (default), "no_change"] need_header_footer_analysis: remove headers and footers from the output result for parsing PDF and images need_binarization: clean pages background (binarize) for PDF without a textual layer and images need_pdf_table_analysis: parse tables for PDF without a textual layer and images delimiter: column separator for CSV, TSV files encoding: encoding of TXT, CSV, TSV )r rrrrrrrrrrrrrrjsonZ return_formatN)superr6rwr3)r"r rwrrrrrrrrrrrrrr __class__r#r'r6s&AzDedocAPIFileLoader.__init__r7ccs0|j|j|j|jd}|j||jdEdHdS)r8)rwr r;r<N) _send_filerwr r3rEr)r"Zdoc_treer#r#r'rG s zDedocAPIFileLoader.lazy_loadcCsiS)Nr#rHr#r#r'r@szDedocAPIFileLoader._make_config)rwr r;r!c Cs~ddl}tj|}t|d(}d||fi}|j|d||d}W5QRX|jdkrjtd|j t |j } | S) z7Send POST-request to `dedoc` API and return the resultsrNrbfilez/upload)filesdatazError during file handling: ) requestsospathbasenameopenpost status_coder5rXdecoderxloads) r"rwr r;r file_namer~rrresultr#r#r'r|s     zDedocAPIFileLoader._send_file)rmrnrorprqrrrrsrr6rr rGrFr@rlistr| __classcell__r#r#rzr'rvjsTM    Urv)rirxrabcrrtypingrrrrrZlangchain_core.documentsr Z)langchain_community.document_loaders.baser r rtrvr#r#r#r's   M