pydantic-evals
Guidelines for evaluating non-deterministic functions with pydantic-evals. Use when writing evals, defining datasets and cases, creating custom evaluators, or testing AI agent outputs with pydantic-evals.
pavelzw · 已有静态检查记录
安装与使用
运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。
npx skills add pavelzw/skill-forge --skill pydantic-evals技能说明
name: pydantic-evals description: >- Guidelines for evaluating non-deterministic functions with pydantic-evals. Use when writing evals, defining datasets and cases, creating custom evaluators, or testing AI agent outputs with pydantic-evals. license: MIT
Evaluating Non-Deterministic Functions with pydantic-evals
pydantic-evals is a code-first framework for evaluating stochastic functions (LLM calls, agents, pipelines). Define test cases, run them against a task function, and score results with evaluators.
Install: pip install pydantic-evals (or pip install 'pydantic-evals[logfire]' for Logfire integration).
Import Reference
from pydantic_evals import Case, Dataset, set_eval_attribute, increment_eval_metric
from pydantic_evals.evaluators import (
Evaluator, EvaluatorContext, EvaluatorOutput, EvaluationReason,
ReportEvaluator, ReportEvaluatorContext,
LLMJudge, HasMatchingSpan,
)
from pydantic_evals.evaluators.common import Equals, EqualsExpected, Contains, IsInstance, MaxDuration
from pydantic_evals.otel import SpanQuery # requires logfire extra
from pydantic_evals.generation import generate_dataset # LLM-based dataset generation
Data Model
Dataset -> Cases -> Evaluators -> EvaluationReport. A Dataset holds Case objects and dataset-wide evaluators. Calling dataset.evaluate(task_fn) runs the task against all cases and returns an EvaluationReport. Both Case and Dataset are generic: Case[InputsT, OutputT, MetadataT].
Case
case = Case(
name="simple", # identifier (optional, but recommended)
inputs="What is the capital of France?", # any type — passed to the task function
expected_output="Paris", # optional — available via ctx.expected_output
metadata={"difficulty": "easy"}, # optional — available via ctx.metadata
evaluators=(MyEvaluator(),), # optional — case-specific evaluators
)
Dataset
dataset = Dataset(
cases=[case1, case2],
evaluators=[GlobalEvaluator()], # applied to every case
report_evaluators=[MyReportEvaluator()], # experiment-wide analysis (optional)
)
| Method | Description |
|---|---|
await dataset.evaluate(task_fn) | Run task against all cases (async) |
dataset.evaluate_sync(task_fn) | Synchronous wrapper |
dataset.add_case(...) | Add a case after construction |
dataset.add_evaluator(ev, specific_case=None) | Add evaluator to all cases or a named case |
Dataset.from_file("cases.yaml") | Load from YAML or JSON |
dataset.to_file("cases.yaml") | Save to YAML or JSON |
EvaluatorContext
Every evaluator receives an EvaluatorContext:
| Field | Type | Description |
|---|---|---|
inputs | InputsT | The case inputs |
output | OutputT | Actual task output |
expected_output | `OutputT | None` |
metadata | `MetadataT | None` |
name | `str | None` |
duration | float | Task execution time in seconds |
span_tree | SpanTree | OpenTelemetry spans recorded during execution |
attributes | dict | Runtime attributes set via set_eval_attribute |
metrics | dict | Runtime metrics set via increment_eval_metric |
Writing Evaluators
Subclass Evaluator and implement evaluate (sync or async). Must use @dataclass decorator.
Return Types
evaluate returns EvaluatorOutput:
bool— pass/fail (stored inReportCase.assertions)int/float— numeric score (stored inReportCase.scores)str— label (stored inReportCase.labels)EvaluationReason(value, reason)— any of the above with an explanationdict[str, ...]— multiple named columns from a single evaluator (see Multi-Score Evaluators)
Single-scalar returns use the evaluator class name as the report column name (override with evaluation_name field).
@dataclass
class ContainsExpected(Evaluator[str, str]):
def evaluate(self, ctx: EvaluatorContext[str, str]) -> EvaluationReason:
if ctx.expected_output is None:
return EvaluationReason(value=False, reason="No expected output provided")
found = ctx.expected_output.lower() in ctx.output.lower()
return EvaluationReason(value=found, reason=f"{'found' if found else 'not found'}")
Built-in Evaluators
| Evaluator | Fields | Description |
|---|---|---|
EqualsExpected() | — | Exact match against expected_output |
Equals(value=...) | value | Exact match against a fixed value |
Contains(value=...) | value, case_sensitive, as_strings | Substring/membership check |
IsInstance(type_name=...) | type_name | Output type check |
MaxDuration(seconds=...) | seconds | Asserts task completed within time limit |
LLMJudge(rubric=...) | rubric, model, include_input, include_expected_output | LLM-based evaluation against a rubric |
HasMatchingSpan(query=...) | query (SpanQuery) | Checks OpenTelemetry span tree for a matching span |
Multi-Score Evaluators (Dict Returns)
When evaluate returns a dict, each key becomes a separate named column in the report. This lets a single evaluator produce multiple independent scores, assertions, or labels from one pass. Values are categorized by type (bool -> assertions, int/float -> scores, str -> labels, EvaluationReason -> unwrapped by inner .value type).
@dataclass
class QualityEvaluator(Evaluator[QAInput, QAOutput]):
"""Single evaluator that produces multiple report columns."""
def evaluate(self, ctx: EvaluatorContext[QAInput, QAOutput]) -> dict[str, EvaluationReason | bool | float]:
output = ctx.output.answer
return {
"is_nonempty": len(output.strip()) > 0,
阅读完整来源 ↗