U h(@s dZddlZddlmZddlmZddlmZddlm Z m Z m Z m Z m Z mZmZmZmZddlmZmZddlmZdd lmZdd lmZmZdd lmZdd lmZm Z dd l!m"Z"m#Z#edZ$ededZ%e e$e e$ge%fe e$dddZ&GdddeZ'dS)zo Ensemble retriever that ensemble the results of multiple retrievers by using weighted Reciprocal Rank Fusion N) defaultdict)Hashable)chain) AnyCallableDictIterableIteratorListOptionalTypeVarcast)#AsyncCallbackManagerForRetrieverRunCallbackManagerForRetrieverRun)Document)root_validator) BaseRetriever RetrieverLike)RunnableConfig) ensure_config patch_config)ConfigurableFieldSpecget_unique_config_specsTH)bound)iterablekeyreturnccs4t}|D]$}||}|kr |||Vq dS)a Yield unique elements of an iterable based on a key function. Args: iterable: The iterable to filter. key: A function that returns a hashable key for each element. Yields: Unique elements of the iterable based on the key function. N)setadd)rrseenekr$A/tmp/pip-unpacked-wheel-bo69hh5q/langchain/retrievers/ensemble.py unique_by_key(s  r&c@sNeZdZUdZeeed<eeed<dZe ed<dZ e e ed<e eedd d Zed d ee efee efd ddZd#e e eeeedddZd$e e eeeedddZe eeedddZe eeedddZdde ee eeedddZdde ee eeedddZeeeeed d!d"ZdS)%EnsembleRetrieveraeRetriever that ensembles the multiple retrievers. It uses a rank fusion. Args: retrievers: A list of retrievers to ensemble. weights: A list of weights corresponding to the retrievers. Defaults to equal weighting for all retrievers. c: A constant added to the rank, controlling the balance between the importance of high-ranked items and the consideration given to lower-ranked items. Default is 60. id_key: The key in the document's metadata used to determine unique documents. If not specified, page_content is used. retrieversweights<cNid_key)rcCstdd|jDS)z+List configurable fields for this runnable.css|]}|jD] }|Vq qdSN) config_specs).0 retrieverspecr$r$r% Qsz1EnsembleRetriever.config_specs..)rr(selfr$r$r%r.NszEnsembleRetriever.config_specsT)pre)valuesrcCs,|ds(t|d}d|g||d<|S)Nr)r()getlen)clsr6Z n_retrieversr$r$r% set_weightsUs  zEnsembleRetriever.set_weights)inputconfigkwargsrc Ksddlm}t|}|j|dd|dd|dg|j|di|jd}|jd|fd |d pj|i|}z|j |||d }Wn0t k r}z| ||W5d}~XYnX|j |f||SdS) Nr)CallbackManager callbacksverboseFtagsmetadatarAZinheritable_tagsZ local_tagsZinheritable_metadataZlocal_metadatanamerun_name run_managerr=) langchain_core.callbacksr?r configurer8rBrCon_retriever_startget_name rank_fusion Exceptionon_retriever_erroron_retriever_end) r4r<r=r>r?callback_managerrHresultr"r$r$r%invoke\s<      zEnsembleRetriever.invokec sddlm}t|}|j|dd|dd|dg|j|di|jd}|jd|fd |d pj|i|IdH}z|j |||d IdH}Wn6t k r}z| |IdH|W5d}~XYnX|j |f|IdH|SdS) Nr)AsyncCallbackManagerr@rAFrBrCrDrErFrG) rIrTrrJr8rBrCrKrL arank_fusionrNrOrP) r4r<r=r>rTrQrHrRr"r$r$r%ainvoke}sD       zEnsembleRetriever.ainvoke)queryrHrcCs|||}|S)z Get the relevant documents for a given query. Args: query: The query to search for. Returns: A list of reranked documents. )rMr4rWrHfused_documentsr$r$r%_get_relevant_documentss z)EnsembleRetriever._get_relevant_documentscs|||IdH}|S)z Asynchronously get the relevant documents for a given query. Args: query: The query to search for. Returns: A list of reranked documents. N)rUrXr$r$r%_aget_relevant_documentssz*EnsembleRetriever._aget_relevant_documents)r=)rWrHr=rcsRfddt|jD}tt|D]}dd||D||<q(||}|S)z Retrieve the results of the retrievers and use rank_fusion_func to get the final result. Args: query: The query to search for. Returns: A list of reranked documents. c s6g|].\}}|tjd|dddqSZ retriever_r7)tag)r@)rSr get_childr/ir0r=rWrHr$r% sz1EnsembleRetriever.rank_fusion..cSs*g|]"}t|tr"ttt|dn|qS) page_content) isinstancestrrr r/docr$r$r%rbs) enumerater(ranger9weighted_reciprocal_rankr4rWrHr=Zretriever_docsr`rYr$rar%rMs   zEnsembleRetriever.rank_fusioncs^tjfddt|jDIdH}tt|D]}dd||D||<q4||}|S)z Asynchronously retrieve the results of the retrievers and use rank_fusion_func to get the final result. Args: query: The query to search for. Returns: A list of reranked documents. c s6g|].\}}|tjd|dddqSr\)rVrr^r_rar$r%rbsz2EnsembleRetriever.arank_fusion..NcSs$g|]}t|tst|dn|qSrc)rerrgr$r$r%rbs)asyncioZgatherrir(rjr9rkrlr$rar%rUs   zEnsembleRetriever.arank_fusion) doc_listsrcst|tjkrtdttt|jD]P\}}t|ddD]:\}}jdkr\|jn |j j||j 7<qBq.t |}t t|fdddfddd }|S) a Perform weighted Reciprocal Rank Fusion on multiple rank lists. You can find more details about RRF here: https://plg.uwaterloo.ca/~gvcormac/cormacksigir09-rrf.pdf Args: doc_lists: A list of rank lists, where each rank list contains unique items. Returns: list: The final aggregated list of items sorted by their weighted RRF scores in descending order. zBsz.Tcs jdkr|jn |jjSr-rprqZ rrf_scorer4r$r%rrGs)reverser)r9r) ValueErrorrfloatziprir,rdrCr+r from_iterablesortedr&)r4rnZdoc_listZweightZrankrhZall_docsZ sorted_docsr$rsr%rks0      z*EnsembleRetriever.weighted_reciprocal_rank)N)N)__name__ __module__ __qualname____doc__r r__annotations__rvr+intr,r rfpropertyrr.rrrr;rrrSrVrrZrr[rMrUrkr$r$r$r%r'9s`    " " %   . , r')(r}rm collectionsrcollections.abcr itertoolsrtypingrrrrr r r r r rIrrZlangchain_core.documentsrZlangchain_core.pydantic_v1rZlangchain_core.retrieversrrZlangchain_core.runnablesrZlangchain_core.runnables.configrrZlangchain_core.runnables.utilsrrrrr&r'r$r$r$r%s    ,     $