U hG@sdZddlmZddlZddlmZmZddlmZddl m Z m Z m Z m Z mZddlmZddlmZdd lmZdd lmZdd lmZeeZGd d d eeZGdddeZGdddZGdddeeZ GdddeeZ!GdddeeZ"dS)z3Interfaces to be implemented by general evaluators.) annotationsN)ABCabstractmethod)Enum)AnyOptionalSequenceTupleUnion)warn) AgentAction)BaseLanguageModel)run_in_executor)Chainc@s`eZdZdZdZdZdZdZdZdZ dZ d Z d Z d Z d Zd ZdZdZdZdZdZdZdZdZdS) EvaluatorTypezThe types of the evaluators.ZqaZcot_qaZ context_qaZpairwise_stringZ score_stringZlabeled_pairwise_stringZlabeled_score_stringZ trajectoryZcriteriaZlabeled_criteriaZstring_distanceZ exact_matchZ regex_matchZpairwise_string_distanceZembedding_distanceZpairwise_embedding_distanceZ json_validityZ json_equalityZjson_edit_distanceZjson_schema_validationN)__name__ __module__ __qualname____doc__ZQAZCOT_QAZ CONTEXT_QAZPAIRWISE_STRINGZ SCORE_STRINGZLABELED_PAIRWISE_STRINGZLABELED_SCORE_STRINGZAGENT_TRAJECTORYZCRITERIAZLABELED_CRITERIAZSTRING_DISTANCEZ EXACT_MATCHZ REGEX_MATCHZPAIRWISE_STRING_DISTANCEZEMBEDDING_DISTANCEZPAIRWISE_EMBEDDING_DISTANCEZ JSON_VALIDITYZ JSON_EQUALITYZJSON_EDIT_DISTANCEZJSON_SCHEMA_VALIDATIONrr?/tmp/pip-unpacked-wheel-bo69hh5q/langchain/evaluation/schema.pyrs,rc@s*eZdZdZeeddddddZdS) LLMEvalChainz,A base class for evaluators that use an LLM.r r)llmkwargsreturncKsdS)z#Create a new evaluator from an LLM.Nr)clsrrrrrfrom_llmMszLLMEvalChain.from_llmN)rrrr classmethodrrrrrrrJsrc@sleZdZdZeddddZeddddZeddd d Zeddd d ZdddddddZ d S)_EvalArgsMixinz(Mixin for checking evaluation arguments.boolrcCsdSz2Whether this evaluator requires a reference label.Frselfrrrrequires_referenceVsz!_EvalArgsMixin.requires_referencecCsdS)0Whether this evaluator requires an input string.Frr"rrrrequires_input[sz_EvalArgsMixin.requires_inputstrcCsd|jjdS)z&Warning to show when input is ignored.zIgnoring input in , as it is not expected. __class__rr"rrr_skip_input_warning`sz"_EvalArgsMixin._skip_input_warningcCsd|jjdS)z*Warning to show when reference is ignored.zIgnoring reference in r(r)r"rrr_skip_reference_warningesz&_EvalArgsMixin._skip_reference_warningN Optional[str]None) referenceinputrcCsx|jr"|dkr"t|jjdn|dk r:|js:t|j|jr\|dkr\t|jjdn|dk rt|jstt|jdS)aCheck if the evaluation arguments are valid. Args: reference (Optional[str], optional): The reference label. input (Optional[str], optional): The input string. Raises: ValueError: If the evaluator requires an input string but none is provided, or if the evaluator requires a reference label but none is provided. Nz requires an input string.z requires a reference string.)r& ValueErrorr*rr r+r$r,)r#r/r0rrr_check_evaluation_argsls z%_EvalArgsMixin._check_evaluation_args)NN) rrrrpropertyr$r&r+r,r2rrrrrSsrc@seZdZdZeddddZeddddZed d d d d d d ddddZd d d d d d d ddddZ d d d dddd ddddZ d d d dddd ddddZ d S)StringEvaluatorzcGrade, tag, or otherwise evaluate predictions relative to their inputs and/or reference labels.r'r cCs|jjS)zThe name of the evaluation.r)r"rrrevaluation_nameszStringEvaluator.evaluation_namercCsdSr!rr"rrrr$sz"StringEvaluator.requires_referenceNr/r0zUnion[str, Any]zOptional[Union[str, Any]]rdict) predictionr/r0rrcKsdS)a:Evaluate Chain or LLM output, based on optional input and label. Args: prediction (str): The LLM or chain prediction to evaluate. reference (Optional[str], optional): The reference label to evaluate against. input (Optional[str], optional): The input to consider during evaluation. kwargs: Additional keyword arguments, including callbacks, tags, etc. Returns: dict: The evaluation results containing the score or value. It is recommended that the dictionary contain the following keys: - score: the score of the evaluation, if applicable. - value: the string value of the evaluation, if applicable. - reasoning: the reasoning for the evaluation, if applicable. Nrr#r8r/r0rrrr_evaluate_stringss z!StringEvaluator._evaluate_stringscs"td|jf|||d|IdHS)aIAsynchronously evaluate Chain or LLM output, based on optional input and label. Args: prediction (str): The LLM or chain prediction to evaluate. reference (Optional[str], optional): The reference label to evaluate against. input (Optional[str], optional): The input to consider during evaluation. kwargs: Additional keyword arguments, including callbacks, tags, etc. Returns: dict: The evaluation results containing the score or value. It is recommended that the dictionary contain the following keys: - score: the score of the evaluation, if applicable. - value: the string value of the evaluation, if applicable. - reasoning: the reasoning for the evaluation, if applicable. Nr8r/r0)rr:r9rrr_aevaluate_stringssz"StringEvaluator._aevaluate_stringsr-cKs&|j||d|jf|||d|S)aEvaluate Chain or LLM output, based on optional input and label. Args: prediction (str): The LLM or chain prediction to evaluate. reference (Optional[str], optional): The reference label to evaluate against. input (Optional[str], optional): The input to consider during evaluation. kwargs: Additional keyword arguments, including callbacks, tags, etc. Returns: dict: The evaluation results containing the score or value. r6r;)r2r:r9rrrevaluate_stringssz StringEvaluator.evaluate_stringscs,|j||d|jf|||d|IdHS)a Asynchronously evaluate Chain or LLM output, based on optional input and label. Args: prediction (str): The LLM or chain prediction to evaluate. reference (Optional[str], optional): The reference label to evaluate against. input (Optional[str], optional): The input to consider during evaluation. kwargs: Additional keyword arguments, including callbacks, tags, etc. Returns: dict: The evaluation results containing the score or value. r6r;N)r2r<r9rrraevaluate_stringssz!StringEvaluator.aevaluate_strings) rrrrr3r5r$rr:r<r=r>rrrrr4s$#r4c @seZdZdZeddddddddddd d Zddddddddddd d Zddddddddddd dZddddddddddddZdS)PairwiseStringEvaluatorzDCompare the output of two models (or two outputs of the same model).Nr6r'r-rr7)r8 prediction_br/r0rrcKsdS)1Evaluate the output string pairs. Args: prediction (str): The output string from the first model. prediction_b (str): The output string from the second model. reference (Optional[str], optional): The expected output / reference string. input (Optional[str], optional): The input string. kwargs: Additional keyword arguments, such as callbacks and optional reference strings. Returns: dict: A dictionary containing the preference, scores, and/or other information. Nrr#r8r@r/r0rrrr_evaluate_string_pairss z.PairwiseStringEvaluator._evaluate_string_pairscs$td|jf||||d|IdHS)@Asynchronously evaluate the output string pairs. Args: prediction (str): The output string from the first model. prediction_b (str): The output string from the second model. reference (Optional[str], optional): The expected output / reference string. input (Optional[str], optional): The input string. kwargs: Additional keyword arguments, such as callbacks and optional reference strings. Returns: dict: A dictionary containing the preference, scores, and/or other information. Nr8r@r/r0)rrCrBrrr_aevaluate_string_pairssz/PairwiseStringEvaluator._aevaluate_string_pairscKs(|j||d|jf||||d|S)rAr6rE)r2rCrBrrrevaluate_string_pairs/sz-PairwiseStringEvaluator.evaluate_string_pairscs.|j||d|jf||||d|IdHS)rDr6rEN)r2rFrBrrraevaluate_string_pairsLsz.PairwiseStringEvaluator.aevaluate_string_pairs) rrrrrrCrFrGrHrrrrr?s#"r?c @seZdZdZeddddZedddd dd d d d ddZdddd dd d d d ddZdddd dd d d d ddZ dddd dd d d d ddZ dS)AgentTrajectoryEvaluatorz,Interface for evaluating agent trajectories.rr cCsdS)r%Trr"rrrr&msz'AgentTrajectoryEvaluator.requires_inputN)r/r'z!Sequence[Tuple[AgentAction, str]]r-rr7)r8agent_trajectoryr0r/rrcKsdS)Evaluate a trajectory. Args: prediction (str): The final predicted response. agent_trajectory (List[Tuple[AgentAction, str]]): The intermediate steps forming the agent trajectory. input (str): The input to the agent. reference (Optional[str]): The reference answer. Returns: dict: The evaluation result. Nrr#r8rJr0r/rrrr_evaluate_agent_trajectoryrs z3AgentTrajectoryEvaluator._evaluate_agent_trajectorycs$td|jf||||d|IdHS)Asynchronously evaluate a trajectory. Args: prediction (str): The final predicted response. agent_trajectory (List[Tuple[AgentAction, str]]): The intermediate steps forming the agent trajectory. input (str): The input to the agent. reference (Optional[str]): The reference answer. Returns: dict: The evaluation result. N)r8rJr/r0)rrMrLrrr_aevaluate_agent_trajectorysz4AgentTrajectoryEvaluator._aevaluate_agent_trajectorycKs(|j||d|jf||||d|S)rKr6r8r0rJr/)r2rMrLrrrevaluate_agent_trajectorysz2AgentTrajectoryEvaluator.evaluate_agent_trajectorycs.|j||d|jf||||d|IdHS)rNr6rPN)r2rOrLrrraevaluate_agent_trajectorysz3AgentTrajectoryEvaluator.aevaluate_agent_trajectory) rrrrr3r&rrMrOrQrRrrrrrIjs%$rI)#r __future__rloggingabcrrenumrtypingrrrr r warningsr Zlangchain_core.agentsr Zlangchain_core.language_modelsr Zlangchain_core.runnables.configrZlangchain.chains.baser getLoggerrloggerr'rrrr4r?rIrrrrs"        6 1tr