U h)@sdZddlmZddlZddlZddlmZmZmZm Z m Z ddl m Z ddl mZddlmZddlmZdd lmZmZmZdd lmZmZdd lmZd d dddZd ddddZGdddeeeZGdddeeeZGdddeZ dS)z-LLM Chains for evaluating question answering.) annotationsN)AnyListOptionalSequenceTuple) Callbacks)BaseLanguageModel)PromptTemplate)LLMChain)CONTEXT_PROMPT COT_PROMPTPROMPT) LLMEvalChainStringEvaluator)RUN_KEYstrzOptional[Tuple[str, int]])textreturncCstd|tj}|rD|ddkr.dS|ddkrDdSz|dt ddt j }|dkrzWdS|dkrWdS|d t ddt j }|dkrWdS|dkrWdSWnt k rYnXdS) Nzgrade:\s*(correct|incorrect)CORRECT)rr INCORRECT)rrr) researchstrip IGNORECASEgroupuppersplit translater maketransstring punctuation IndexError)rmatchZ first_word last_wordr(F/tmp/pip-unpacked-wheel-bo69hh5q/langchain/evaluation/qa/eval_chain.py _get_scores4       r*dictcCs6|}t|}|dkr"d\}}n|\}}|||dS)zParse the output text. Args: text (str): The output text to parse. Returns: Any: The parsed output. N)NN) reasoningvaluescore)rr*)rr,Z parsed_scoresr-r.r(r(r)_parse_string_eval_output0s  r/c @s eZdZUdZdZded<GdddZeddd d Ze ddd d Z e ddd dZ e ddddZ ed.dddddddZ d/ddddddddddd d!Zd"d"d#d$d%Zdddd&d'dd(d(dddd"d)d*d+Zdddd&d'dd(d(dddd"d)d,d-ZdS)0 QAEvalChainz,LLM Chain for evaluating question answering.resultsr output_keyc@seZdZdZdS)zQAEvalChain.ConfigignoreN__name__ __module__ __qualname__extrar(r(r(r)ConfigKsr9boolrcCsdSNFr(clsr(r(r)is_lc_serializableNszQAEvalChain.is_lc_serializablecCsdS)NZ correctnessr(selfr(r(r)evaluation_nameRszQAEvalChain.evaluation_namecCsdSNTr(r@r(r(r)requires_referenceVszQAEvalChain.requires_referencecCsdSrCr(r@r(r(r)requires_inputZszQAEvalChain.requires_inputNr Optional[PromptTemplate]rllmpromptkwargsrcKsJ|pt}dddh}|t|jkr6td|d|j|f||d|S)aLoad QA Eval Chain from LLM. Args: llm (BaseLanguageModel): the base language model to use. prompt (PromptTemplate): A prompt template containing the input_variables: 'input', 'answer' and 'result' that will be used as the prompt for evaluation. Defaults to PROMPT. **kwargs: additional keyword arguments. Returns: QAEvalChain: the loaded QA eval chain. queryanswerresultInput variables should be , but got rHrI)rsetinput_variables ValueError)r>rHrIrJexpected_input_varsr(r(r)from_llm^s zQAEvalChain.from_llmrKrLrM callbackszSequence[dict]r List[dict])examples predictions question_key answer_keyprediction_keyrWrcs*fddt|D}|j||dS)5Evaluate question answering examples and predictions.cs,g|]$\}}|||dqS)rKrLrMr(.0iZexampler\r]rZr[r(r) s  z(QAEvalChain.evaluate..rV enumerateapply)rArYrZr[r\r]rWinputsr(rcr)evaluate}s  zQAEvalChain.evaluater+rMrcCs&t||j}t|kr"|t|t<|SNr/r2rrArM parsed_resultr(r(r)_prepare_outputs zQAEvalChain._prepare_outputF referenceinputrWinclude_run_info Optional[str] predictionrqrrrWrsrJrcKs ||||d||d}||S)aEvaluate Chain or LLM output, based on optional input and label. Args: prediction (str): the LLM or chain prediction to evaluate. reference (Optional[str], optional): the reference label to evaluate against. input (Optional[str], optional): the input to consider during evaluation callbacks (Callbacks, optional): the callbacks to use for tracing. include_run_info (bool, optional): whether to include run info in the returned results. **kwargs: additional keyword arguments, including callbacks, tags, etc. Returns: dict: The evaluation results containing the score or value. r_rWrsrorArvrqrrrWrsrJrMr(r(r)_evaluate_stringss zQAEvalChain._evaluate_stringscs(|j|||d||dIdH}||S)Nr_rhrWrsZacallroryr(r(r)_aevaluate_stringss   zQAEvalChain._aevaluate_strings)N)rKrLrM)r5r6r7__doc__r2__annotations__r9 classmethodr?propertyrBrDrErUrirorzr}r(r(r(r)r0Fs>  " 'r0c @seZdZdZeddddZeddddZedddd ZGd d d Z ed d dddZ eddddZ ed0dddddddZ d1dddddddd dd!d"d#Z d$d$d%d&d'Zdddd(d)dd*d*d ddd$d+d,d-Zdddd(d)dd*d*d ddd$d+d.d/ZdS)2ContextQAEvalChainz3LLM Chain for evaluating QA w/o GT based on contextr:r;cCsdSr<r(r=r(r(r)r?sz%ContextQAEvalChain.is_lc_serializablecCsdS)z.Whether the chain requires a reference string.Tr(r@r(r(r)rDsz%ContextQAEvalChain.requires_referencecCsdS)z+Whether the chain requires an input string.Tr(r@r(r(r)rEsz!ContextQAEvalChain.requires_inputc@seZdZdZdS)zContextQAEvalChain.Configr3Nr4r(r(r(r)r9sr9r None)rIrcCs2dddh}|t|jkr.td|d|jdS)NrKcontextrMrNrO)rQrRrS)r>rIrTr(r(r)_validate_input_varss  z'ContextQAEvalChain._validate_input_varsrcCsdS)NzContextual Accuracyr(r@r(r(r)rBsz"ContextQAEvalChain.evaluation_nameNr rFrrGcKs&|pt}|||f||d|S)aLoad QA Eval Chain from LLM. Args: llm (BaseLanguageModel): the base language model to use. prompt (PromptTemplate): A prompt template containing the input_variables: 'query', 'context' and 'result' that will be used as the prompt for evaluation. Defaults to PROMPT. **kwargs: additional keyword arguments. Returns: ContextQAEvalChain: the loaded QA eval chain. rP)r rr>rHrIrJr(r(r)rUs zContextQAEvalChain.from_llmrKrrMrVrXr)rYrZr[ context_keyr]rWrcs*fddt|D}|j||dS)r^cs,g|]$\}}|||dqS)rKrrMr(r`rr]rZr[r(r)rds  z/ContextQAEvalChain.evaluate..rVre)rArYrZr[rr]rWrhr(rr)ri s  zContextQAEvalChain.evaluater+rjcCs&t||j}t|kr"|t|t<|Srkrlrmr(r(r)ros z"ContextQAEvalChain._prepare_outputFrprtrucKs ||||d||d}||S)Nrrwrxryr(r(r)rz%s  z$ContextQAEvalChain._evaluate_stringscs(|j|||d||dIdH}||S)Nrr{r|ryr(r(r)r}:s   z%ContextQAEvalChain._aevaluate_strings)N)rKrrM)r5r6r7r~rr?rrDrEr9rrBrUrirorzr}r(r(r(r)rs@ rc@sNeZdZdZeddddZeddddZedd d d dd ddZd S)CotQAEvalChainz=LLM Chain for evaluating QA using chain of thought reasoning.r:r;cCsdSr<r(r=r(r(r)r?Osz!CotQAEvalChain.is_lc_serializablercCsdS)NzCOT Contextual Accuracyr(r@r(r(r)rBSszCotQAEvalChain.evaluation_nameNr rFrrGcKs&|pt}|||f||d|S)zLoad QA Eval Chain from LLM.rP)r rrr(r(r)rUWs zCotQAEvalChain.from_llm)N) r5r6r7r~rr?rrBrUr(r(r(r)rLsr)!r~ __future__rrr#typingrrrrrZ langchain_core.callbacks.managerrZlangchain_core.language_modelsr Zlangchain_core.promptsr Zlangchain.chains.llmr Z#langchain.evaluation.qa.eval_promptr r rZlangchain.evaluation.schemarrZlangchain.schemarr*r/r0rrr(r(r(r)s"       ~