跳至正文
OpenAgentSkill
浏览已开放 · 目录记录日期 2026-11-21 · 自动更新和账户服务暂时关闭。
开发与测试

pydantic-evals

Guidelines for evaluating non-deterministic functions with pydantic-evals. Use when writing evals, defining datasets and cases, creating custom evaluators, or testing AI agent outputs with pydantic-evals.

pavelzw · 已有静态检查记录

安装与使用

运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。

npx skills add pavelzw/skill-forge --skill pydantic-evals

技能说明


name: pydantic-evals description: >- Guidelines for evaluating non-deterministic functions with pydantic-evals. Use when writing evals, defining datasets and cases, creating custom evaluators, or testing AI agent outputs with pydantic-evals. license: MIT

Evaluating Non-Deterministic Functions with pydantic-evals

pydantic-evals is a code-first framework for evaluating stochastic functions (LLM calls, agents, pipelines). Define test cases, run them against a task function, and score results with evaluators.

Install: pip install pydantic-evals (or pip install 'pydantic-evals[logfire]' for Logfire integration).

Import Reference

from pydantic_evals import Case, Dataset, set_eval_attribute, increment_eval_metric
from pydantic_evals.evaluators import (
    Evaluator, EvaluatorContext, EvaluatorOutput, EvaluationReason,
    ReportEvaluator, ReportEvaluatorContext,
    LLMJudge, HasMatchingSpan,
)
from pydantic_evals.evaluators.common import Equals, EqualsExpected, Contains, IsInstance, MaxDuration
from pydantic_evals.otel import SpanQuery               # requires logfire extra
from pydantic_evals.generation import generate_dataset  # LLM-based dataset generation

Data Model

Dataset -> Cases -> Evaluators -> EvaluationReport. A Dataset holds Case objects and dataset-wide evaluators. Calling dataset.evaluate(task_fn) runs the task against all cases and returns an EvaluationReport. Both Case and Dataset are generic: Case[InputsT, OutputT, MetadataT].

Case

case = Case(
    name="simple",                           # identifier (optional, but recommended)
    inputs="What is the capital of France?", # any type — passed to the task function
    expected_output="Paris",                 # optional — available via ctx.expected_output
    metadata={"difficulty": "easy"},         # optional — available via ctx.metadata
    evaluators=(MyEvaluator(),),             # optional — case-specific evaluators
)

Dataset

dataset = Dataset(
    cases=[case1, case2],
    evaluators=[GlobalEvaluator()],          # applied to every case
    report_evaluators=[MyReportEvaluator()], # experiment-wide analysis (optional)
)
MethodDescription
await dataset.evaluate(task_fn)Run task against all cases (async)
dataset.evaluate_sync(task_fn)Synchronous wrapper
dataset.add_case(...)Add a case after construction
dataset.add_evaluator(ev, specific_case=None)Add evaluator to all cases or a named case
Dataset.from_file("cases.yaml")Load from YAML or JSON
dataset.to_file("cases.yaml")Save to YAML or JSON

EvaluatorContext

Every evaluator receives an EvaluatorContext:

FieldTypeDescription
inputsInputsTThe case inputs
outputOutputTActual task output
expected_output`OutputTNone`
metadata`MetadataTNone`
name`strNone`
durationfloatTask execution time in seconds
span_treeSpanTreeOpenTelemetry spans recorded during execution
attributesdictRuntime attributes set via set_eval_attribute
metricsdictRuntime metrics set via increment_eval_metric

Writing Evaluators

Subclass Evaluator and implement evaluate (sync or async). Must use @dataclass decorator.

Return Types

evaluate returns EvaluatorOutput:

  • bool — pass/fail (stored in ReportCase.assertions)
  • int/float — numeric score (stored in ReportCase.scores)
  • str — label (stored in ReportCase.labels)
  • EvaluationReason(value, reason) — any of the above with an explanation
  • dict[str, ...] — multiple named columns from a single evaluator (see Multi-Score Evaluators)

Single-scalar returns use the evaluator class name as the report column name (override with evaluation_name field).

@dataclass
class ContainsExpected(Evaluator[str, str]):
    def evaluate(self, ctx: EvaluatorContext[str, str]) -> EvaluationReason:
        if ctx.expected_output is None:
            return EvaluationReason(value=False, reason="No expected output provided")
        found = ctx.expected_output.lower() in ctx.output.lower()
        return EvaluationReason(value=found, reason=f"{'found' if found else 'not found'}")

Built-in Evaluators

EvaluatorFieldsDescription
EqualsExpected()—Exact match against expected_output
Equals(value=...)valueExact match against a fixed value
Contains(value=...)value, case_sensitive, as_stringsSubstring/membership check
IsInstance(type_name=...)type_nameOutput type check
MaxDuration(seconds=...)secondsAsserts task completed within time limit
LLMJudge(rubric=...)rubric, model, include_input, include_expected_outputLLM-based evaluation against a rubric
HasMatchingSpan(query=...)query (SpanQuery)Checks OpenTelemetry span tree for a matching span

Multi-Score Evaluators (Dict Returns)

When evaluate returns a dict, each key becomes a separate named column in the report. This lets a single evaluator produce multiple independent scores, assertions, or labels from one pass. Values are categorized by type (bool -> assertions, int/float -> scores, str -> labels, EvaluationReason -> unwrapped by inner .value type).

@dataclass
class QualityEvaluator(Evaluator[QAInput, QAOutput]):
    """Single evaluator that produces multiple report columns."""

    def evaluate(self, ctx: EvaluatorContext[QAInput, QAOutput]) -> dict[str, EvaluationReason | bool | float]:
        output = ctx.output.answer
        return {
            "is_nonempty": len(output.strip()) > 0,                      
阅读完整来源 ↗