U hX@sxddlmZddlZddlmZmZmZmZmZddl m Z m Z Gddde Z dddd d d d Z Gd dde ZdS)) annotationsN)AnyListLiteralOptionalUnion)Language TextSplittercs>eZdZdZddddddfd d Zdd d d dZZS)CharacterTextSplitterz(Splitting text that looks at characters. FstrboolrNone) separatoris_separator_regexkwargsreturnc stjf|||_||_dS)Create a new TextSplitter.N)super__init__ _separator_is_separator_regex)selfrrr __class__F/tmp/pip-unpacked-wheel-a648t6hw/langchain_text_splitters/character.pyr szCharacterTextSplitter.__init__ List[str]textrcCsB|jr |jn t|j}t|||j}|jr0dn|j}|||S)&Split incoming text and return chunks.)rrreescape_split_text_with_regex_keep_separator _merge_splits)rrrsplitsrrrr split_texts z CharacterTextSplitter.split_text)r F)__name__ __module__ __qualname____doc__rr( __classcell__rrrrr s r r &Union[bool, Literal[('start', 'end')]]r)rrkeep_separatorrcs|r|rtd|d||dkrFfddtdtddDnfd dtdtdD}tddkr|d d7}|dkr|d gn dg|}qt||}nt|}d d|DS) N()endcs g|]}||dqSr.0iZ_splitsrr (sz*_split_text_with_regex..rr4cs g|]}||dqSr3rr5r8rrr9*scSsg|]}|dkr|qS)r!r)r6srrrr97s)r"splitrangelenlist)rrr/r'rr8rr$s $ r$cs|eZdZdZdddddd d fd d Zd dddddZd ddddZeddddddZe dddddZ Z S)RecursiveCharacterTextSplitterzSplitting text by recursively look at characters. Recursively tries to split by different characters to find one that works. NTFzOptional[List[str]]r.r rr) separatorsr/rrrc s2tjfd|i||p$ddddg|_||_dS)rr/r   r!N)rr _separatorsr)rrBr/rrrrrrAsz'RecursiveCharacterTextSplitter.__init__r r)rrBrcCs(g}|d}g}t|D]P\}}|jr*|nt|}|dkrD|}qjt||r|}||dd}qjq|jrt|nt|}t|||j} g} |jrdn|}| D]d} || |jkr| | q| r| | |} | | g} |s| | q| | |} | | q| r$| | |} | | |S)r r;r!r4N) enumeraterr"r#searchr$r%Z_length_functionZ _chunk_sizeappendr&extend _split_text)rrrBZ final_chunksrZnew_separatorsr7Z_srr'Z _good_splitsr<Z merged_textZ other_inforrrrJMs>         z*RecursiveCharacterTextSplitter._split_textrcCs|||jS)N)rJrE)rrrrrr(usz)RecursiveCharacterTextSplitter.split_textr)languagerrcKs||}|f|dd|S)NT)rBr)get_separators_for_language)clsrKrrBrrr from_languagexs z,RecursiveCharacterTextSplitter.from_language)rKrcCs|tjks|tjkr4ddddddddd d d d d dgS|tjkrZddddddd d d d d dg S|tjkrddddddddd d d d d dgS|tjkrdddddddddddddd dd d d dgS|tjkrddddddddd d dd d d dgS|tjkrd d!d"dddddddddd d dd d d dgS|tjkrBdddd#dd$d d d d d dg S|tj krfd%d&d d'd(d)d d d dg S|tj krdd*d+d d d dgS|tj krd,d-d.d/d d d dgS|tj krd*ddd0ddd$d1d2d d d dg S|tj krd*d3d4d5d6d7dd0dd d8d9dd$d d d dgS|tjkr,d:dddddd;dd dddd$d d d d d dgS|tjkrd?d@dAdBdCd d d dg S|tjkrdDdEdFdGdHdIdJdKdLdMdNdOdPdQdRd dgS|tjkrdSdTdUdVdWdXdYdZd[d\d]d^d_d`dadbdcdddedfdgdhdidjdkdldgS|tjkrhd!d dmdndoddpdddddqddrdd#dd dsd ddtdudvdwd d d dgS|tjkrdxdydzd!d{d|dddod}d~d>d dddddd d d dgS|tjkrdddddddddddddddddddddd d dgS|tjkrddddddd d d dg S|tjkrNddddd$ddddddddd(ddddd ddddd d d dgS|tjkr|dddd#ddd ddtdwdvd d d dgS|tjkrtd|dntd|dttdS)Nz class z void z int z float z double z if z for z while z switch z case r rCrDr!z func z var z const z type z public z protected z private z static z internal z companion z fun z val z when z else z function z let z default z enum z interface z namespace z foreach z do z message z service z option z import z syntax z def z def z =+ z -+ z \*+ z .. * z unless z begin z rescue z defp z defmodule z defprotocol z defmacro z defmacrop z cond z with z fn z loop z match z object z struct z #{1,6} z``` z \*\*\*+ z ---+ z ___+ z \\chapter{z \\section{z \\subsection{z \\subsubsection{z \\begin{enumerate}z \\begin{itemize}z \\begin{description}z \\begin{list}z \\begin{quote}z \\begin{quotation}z \\begin{verse}z \\begin{verbatim}z \egin{align}z$$$zs