U h&@stddlmZddlZddlmZddlmZmZmZm Z m Z ddl m Z ddl mZddlmZGdd d eZdS) ) annotationsN)Path)AnyDictIterableListOptional)CallbackManagerForRetrieverRunDocument) BaseRetrieverc@seZdZUdZded<ded<ded<dZded <Gd d d Zed(d dddddddZed ddddddddZ ddddddZ d)ddddd d!Z ed"dd#dd$ddd%d&d'Z d S)*TFIDFRetrieverz`TF-IDF` retriever. Largely based on https://github.com/asvskartheek/Text-Retrieval/blob/master/TF-IDF%20Search%20Engine%20(SKLEARN).ipynb r vectorizerzList[Document]docs tfidf_arrayintkc@seZdZdZdS)zTFIDFRetriever.ConfigTN)__name__ __module__ __qualname__Zarbitrary_types_allowedrrH/tmp/pip-unpacked-wheel-9gdii04g/langchain_community/retrievers/tfidf.pyConfigsrNz Iterable[str]zOptional[Iterable[dict]]zOptional[Dict[str, Any]])texts metadatas tfidf_paramskwargsreturnc Kszddlm}Wntk r,tdYnX|p4i}|f|}||}|pZdd|D}ddt||D}|f|||d|S) Nr)TfidfVectorizerzNCould not import scikit-learn, please install with `pip install scikit-learn`.css|] }iVqdSNr).0_rrr 2sz,TFIDFRetriever.from_texts..cSsg|]\}}t||dqS)Z page_contentmetadatar )r!tmrrr 3sz-TFIDFRetriever.from_texts..rrr)Zsklearn.feature_extraction.textr ImportErrorZ fit_transformzip) clsrrrrrrrrrrr from_textss   zTFIDFRetriever.from_texts)rzIterable[Document]) documentsrrrcKs.tdd|D\}}|jf|||d|S)Ncss|]}|j|jfVqdSr r$)r!drrrr#>sz0TFIDFRetriever.from_documents..)rrr)r+r-)r,r.rrrrrrrfrom_documents6szTFIDFRetriever.from_documentsstrr )query run_managerrcs\ddlm}j|g}|j|d}fdd|j ddddD}|S)Nr)cosine_similarity)csg|]}j|qSr)r)r!iselfrrr(Nsz:TFIDFRetriever._get_relevant_documents..r5)Zsklearn.metrics.pairwiser4rZ transformrZreshapeZargsortr)r8r2r3r4Z query_vecresultsZ return_docsrr7r_get_relevant_documentsCs  ,z&TFIDFRetriever._get_relevant_documentstfidf_vectorizerNone) folder_path file_namerc Csz ddl}Wntk r(tdYnXt|}|jddd||j||dt||dd}t|j|j f|W5QRXdS)NrBCould not import joblib, please install with `pip install joblib`.T)exist_okparents.joblib.pklwb) joblibr*rmkdirdumpropenpicklerr)r8r=r>rEpathfrrr save_localQs  zTFIDFRetriever.save_localF)allow_dangerous_deserializationr>bool)r=rMr>rc Csz ddl}Wntk r(tdYnX|s6tdt|}|||d}t||dd}t|\}} W5QRX|||| dS) aLoad the retriever from local storage. Args: folder_path: Folder path to load from. allow_dangerous_deserialization: Whether to allow dangerous deserialization. Defaults to False. The deserialization relies on .joblib and .pkl files, which can be modified to deliver a malicious payload that results in execution of arbitrary code on your machine. You will need to set this to `True` to use deserialization. If you do this, make sure you trust the source of the file. file_name: File name to load from. Defaults to "tfidf_vectorizer". Returns: TFIDFRetriever: Loaded retriever. rNr?aThe de-serialization of this retriever is based on .joblib and .pkl files.Such files can be modified to deliver a malicious payload that results in execution of arbitrary code on your machine.You will need to set `allow_dangerous_deserialization` to `True` to load this retriever. If you do this, make sure you trust the source of the file, and you are responsible for validating the file came from a trusted source.rBrCrbr))rEr* ValueErrorrloadrHrI) r,r=rMr>rErJrrKrrrrr load_localgs   zTFIDFRetriever.load_local)NN)r;) rrr__doc____annotations__rr classmethodr-r0r:rLrRrrrrr s(   r ) __future__rrIpathlibrtypingrrrrrZlangchain_core.callbacksr Zlangchain_core.documentsr Zlangchain_core.retrieversr r rrrrs