U hg4@sPdZddlmZmZmZmZmZmZmZddl m Z ddl m Z ddl mZddlmZmZddlmZddlmZmZdd lmZmZdd lmZdd lmZdd lm Z m!Z!dd l"m#Z$eeeegeeee%ffZ&eeeeeegeeee%ffZ'GdddeZ(Gddde(Z)ee&ee!fZ*ee e+e(fZ,GdddeZ-dS)z!Configuration for run evaluators.)AnyCallableDictListOptionalSequenceUnion) Embeddings)BaseLanguageModel)BasePromptTemplate) BaseModelField) RunEvaluator)EvaluationResultEvaluationResults)ExampleRun) CRITERIA_TYPE)EmbeddingDistance) EvaluatorTypeStringEvaluator)StringDistancec@s0eZdZUdZeed<eeefdddZ dS) EvalConfigaConfiguration for a given run evaluator. Parameters ---------- evaluator_type : EvaluatorType The type of evaluator to use. Methods ------- get_kwargs() Get the keyword arguments for the evaluator configuration. evaluator_typereturncCs4i}|D]&\}}|dkrqn |dkr&q|||<q|S)zGet the keyword arguments for the load_evaluator call. Returns ------- Dict[str, Any] The keyword arguments for the load_evaluator call. rN)selfkwargsfieldvalrrE/tmp/pip-unpacked-wheel-bo69hh5q/langchain/smith/evaluation/config.py get_kwargs0s   zEvalConfig.get_kwargsN) __name__ __module__ __qualname____doc__r__annotations__rstrrr"rrrr!rs rcs`eZdZUdZdZeeed<dZeeed<dZ eeed<e ee fdfdd Z Z S) SingleKeyEvalConfigzBConfiguration for a run evaluator that only requires a single key.N reference_keyprediction_key input_keyrcs$t}dD]}||dq|S)N)r*r+r,)superr"pop)rrkey __class__rr!r"Qs zSingleKeyEvalConfig.get_kwargs)r#r$r%r&r*rr(r'r+r,rrr" __classcell__rrr0r!r)Cs r)c@sveZdZUdZeedZeee e fe d<dZ e ee e d<dZe eee d<dZe ee d<dZe ee d<dZe ee d <dZe ee d <Gd d d ZGd ddeZGdddeZGdddeZGdddeZGdddeZGdddeZGdddeZGdddeZGddde Z!Gdd d eZ"Gd!d"d"eZ#Gd#d$d$eZ$Gd%d&d&e$Z%dS)' RunEvalConfiga0Configuration for a run evaluation. Parameters ---------- evaluators : List[Union[EvaluatorType, EvalConfig, RunEvaluator, Callable]] Configurations for which evaluators to apply to the dataset run. Each can be the string of an :class:`EvaluatorType `, such as EvaluatorType.QA, the evaluator type string ("qa"), or a configuration for a given evaluator (e.g., :class:`RunEvalConfig.QA `). custom_evaluators : Optional[List[Union[RunEvaluator, StringEvaluator]]] Custom evaluators to apply to the dataset run. reference_key : Optional[str] The key in the dataset run to use as the reference string. If not provided, it will be inferred automatically. prediction_key : Optional[str] The key from the traced run's outputs dictionary to use to represent the prediction. If not provided, it will be inferred automatically. input_key : Optional[str] The key from the traced run's inputs dictionary to use to represent the input. If not provided, it will be inferred automatically. eval_llm : Optional[BaseLanguageModel] The language model to pass to any evaluators that use a language model. )default_factory evaluatorsNcustom_evaluatorsbatch_evaluatorsr*r+r,eval_llmc@seZdZdZdS)zRunEvalConfig.ConfigTNr#r$r%Zarbitrary_types_allowedrrrr!Configsr:cs`eZdZUdZdZeeed<dZee ed<e j Z e ed<d eee ddfdd ZZS) zRunEvalConfig.Criteriaa"Configuration for a reference-free criteria evaluator. Parameters ---------- criteria : Optional[CRITERIA_TYPE] The criteria to evaluate. llm : Optional[BaseLanguageModel] The language model to use for the evaluation chain. Ncriteriallmrr;rrc stjfd|i|dSNr;r-__init__rr;rr0rr!r@szRunEvalConfig.Criteria.__init__)N)r#r$r%r&r;rrr'r<r rZCRITERIArrr@r2rrr0r!Criterias  rBcs`eZdZUdZdZeeed<dZee ed<e j Z e ed<d eee ddfdd ZZS) zRunEvalConfig.LabeledCriteriaa,Configuration for a labeled (with references) criteria evaluator. Parameters ---------- criteria : Optional[CRITERIA_TYPE] The criteria to evaluate. llm : Optional[BaseLanguageModel] The language model to use for the evaluation chain. Nr;r<rr=c stjfd|i|dSr>r?rAr0rr!r@sz&RunEvalConfig.LabeledCriteria.__init__)N)r#r$r%r&r;rrr'r<r rZLABELED_CRITERIArrr@r2rrr0r!LabeledCriterias  rCc@sNeZdZUdZejZeed<dZe e ed<dZ e e ed<GdddZ dS)zRunEvalConfig.EmbeddingDistanceaIConfiguration for an embedding distance evaluator. Parameters ---------- embeddings : Optional[Embeddings] The embeddings to use for computing the distance. distance_metric : Optional[EmbeddingDistanceEnum] The distance metric to use for computing the distance. rN embeddingsdistance_metricc@seZdZdZdS)z&RunEvalConfig.EmbeddingDistance.ConfigTNr9rrrr!r:sr:)r#r$r%r&rZEMBEDDING_DISTANCErr'rDrr rEEmbeddingDistanceEnumr:rrrr!rs  rc@s<eZdZUdZejZeed<dZe e ed<dZ e ed<dS)zRunEvalConfig.StringDistancezConfiguration for a string distance evaluator. Parameters ---------- distance : Optional[StringDistanceEnum] The string distance metric to use. rNdistanceTnormalize_score) r#r$r%r&rZSTRING_DISTANCErr'rGrStringDistanceEnumrHboolrrrr!rs   rc@s@eZdZUdZejZeed<dZe e ed<dZ e e ed<dS)zRunEvalConfig.QAa-Configuration for a QA evaluator. Parameters ---------- prompt : Optional[BasePromptTemplate] The prompt template to use for generating the question. llm : Optional[BaseLanguageModel] The language model to use for the evaluation chain. rNr<prompt) r#r$r%r&rQArr'r<rr rKr rrrr!rLs  rLc@s@eZdZUdZejZeed<dZe e ed<dZ e e ed<dS)zRunEvalConfig.ContextQA<Configuration for a context-based QA evaluator. Parameters ---------- prompt : Optional[BasePromptTemplate] The prompt template to use for generating the question. llm : Optional[BaseLanguageModel] The language model to use for the evaluation chain. rNr<rK r#r$r%r&rZ CONTEXT_QArr'r<rr rKr rrrr! ContextQAs  rOc@s@eZdZUdZejZeed<dZe e ed<dZ e e ed<dS)zRunEvalConfig.CoTQArMrNr<rKrNrrrr!CoTQAs  rPc@s eZdZUdZejZeed<dS)zRunEvalConfig.JsonValidityz\Configuration for a json validity evaluator. Parameters ---------- rN)r#r$r%r&rZ JSON_VALIDITYrr'rrrr! JsonValidity$s rQc@s eZdZUdZejZeed<dS)z#RunEvalConfig.JsonEqualityEvaluatorz\Configuration for a json equality evaluator. Parameters ---------- rN)r#r$r%r&rZ JSON_EQUALITYrr'rrrr!JsonEqualityEvaluator-s rRc@sDeZdZUdZejZeed<dZe ed<dZ e ed<dZ e ed<dS)zRunEvalConfig.ExactMatchaxConfiguration for an exact match string evaluator. Parameters ---------- ignore_case : bool Whether to ignore case when comparing strings. ignore_punctuation : bool Whether to ignore punctuation when comparing strings. ignore_numbers : bool Whether to ignore numbers when comparing strings. rF ignore_caseignore_punctuationignore_numbersN) r#r$r%r&rZ EXACT_MATCHrr'rSrJrTrUrrrr! ExactMatch6s    rVc@s,eZdZUdZejZeed<dZe ed<dS)zRunEvalConfig.RegexMatchzConfiguration for a regex match string evaluator. Parameters ---------- flags : int The flags to pass to the regex. Example: re.IGNORECASE. rrflagsN) r#r$r%r&rZ REGEX_MATCHrr'rWintrrrr! RegexMatchHs rYcseZdZUdZejZeed<dZe e ed<dZ e e ed<dZ e eed<dZe eed<d e e e eeddfd d ZZS) zRunEvalConfig.ScoreStringaConfiguration for a score string evaluator. This is like the criteria evaluator but it is configured by default to return a score on the scale from 1-10. It is recommended to normalize these scores by setting `normalize_by` to 10. Parameters ---------- criteria : Optional[CRITERIA_TYPE] The criteria to evaluate. llm : Optional[BaseLanguageModel] The language model to use for the evaluation chain. normalize_by: Optional[int] = None If you want to normalize the score, the denominator to use. If not provided, the score will be between 1 and 10 (by default). prompt : Optional[BasePromptTemplate] rNr;r< normalize_byrK)r;rZrrc stjf||d|dS)N)r;rZr?)rr;rZrr0rr!r@osz"RunEvalConfig.ScoreString.__init__)NN)r#r$r%r&rZ SCORE_STRINGrr'r;rrr<r rZfloatrKr rr@r2rrr0r! ScoreStringTs r\c@seZdZUejZeed<dS)z RunEvalConfig.LabeledScoreStringrN)r#r$r%rZLABELED_SCORE_STRINGrr'rrrr!LabeledScoreStringws r])&r#r$r%r&r listr5rrSINGLE_EVAL_CONFIG_TYPECUSTOM_EVALUATOR_TYPEr'r6rr7BATCH_EVALUATOR_LIKEr*r(r+r,r8r r:r)rBrCrrrLrOrPrQrrRrVrYr\r]rrrr!r3]s: #    #r3N).r&typingrrrrrrrZlangchain_core.embeddingsr Zlangchain_core.language_modelsr Zlangchain_core.promptsr Zlangchain_core.pydantic_v1r r Z langsmithrZlangsmith.evaluation.evaluatorrrZlangsmith.schemasrrZ(langchain.evaluation.criteria.eval_chainrZ,langchain.evaluation.embedding_distance.baserrFZlangchain.evaluation.schemarrZ)langchain.evaluation.string_distance.baserrIdictZRUN_EVALUATOR_LIKErarr)r`r(r_r3rrrr!s2$        $