eval-code
Write deterministic, code-based evaluator functions for an AI product. Use this skill whenever you need to write evaluators that check structural properties of AI outputs (format, count, presence, schema compliance) without calling another LLM. Also use when auditing existing code-based evaluators for brittleness, coverage gaps, or false positives.
Calibre-Labs · 已有静态检查记录
安装与使用
运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。
npx skills add Calibre-Labs/reforge-ai-evals --skill eval-code技能说明
name: eval-code description: > Write deterministic, code-based evaluator functions for an AI product. Use this skill whenever you need to write evaluators that check structural properties of AI outputs (format, count, presence, schema compliance) without calling another LLM. Also use when auditing existing code-based evaluators for brittleness, coverage gaps, or false positives.
Code-Based Evaluator Skill
Code-based evaluators are deterministic functions that check AI outputs using regex, string operations, schema validation, or simple parsing — no LLM call required. They are fast, cheap, and produce the same result every time, which makes them the foundation of any eval suite.
The tradeoff: they can only check what is mechanically verifiable. They cannot assess whether the content is correct, relevant, or well-reasoned. Use them for structural properties; use LLM judges for semantic ones.
When to use this skill
Write code-based evaluators when the property you want to check can be expressed as:
- Presence/absence: does the output contain X? (a section heading, a citation, a table)
- Count: does the output contain exactly/at least N of something? (3 companies, 3 sources)
- Format compliance: is the output in the expected structure? (JSON schema, markdown table)
- Value range: is a number within expected bounds? (confidence score 0–1, year 1900–2030)
- String match: does the output contain a required string? (company name, required disclaimer)
Do NOT write a code-based evaluator for:
- Whether the companies ranked are the right companies (use an LLM judge)
- Whether the rationale is coherent (use an LLM judge)
- Whether citations are accurate (requires external lookup or LLM judge)
Workflow
Step 1 — Identify the property to check
Start from the product's output requirements, not from what seems easy to check. Ask: "What must always be true in a valid output, regardless of the input?"
For each property, write one sentence: "A valid output must [verb] [object]."
Examples:
- A valid output must contain exactly N ranked results.
- A valid output must include at least 3 source citations.
- A valid output must include a summary section before the detailed breakdown.
- A valid output must be parseable as valid JSON matching the schema.
If the sentence requires words like "correctly" or "appropriate" or "relevant", stop — that property needs an LLM judge, not a code check.
Step 2 — Write the function signature
The standard signature works across most eval platforms (Braintrust, LangSmith, custom runners):
def evaluator_name(output: str, input: str, expected: str = None) -> float | dict:
...
Return a float between 0.0 and 1.0, or a dict with at least a score key:
{"score": 0.5, "metadata": {"reason": "found 2 citations, expected 3"}}
Return metadata whenever partial credit is possible — it tells you why a score is 0.5, not just that it is.
Step 3 — Write the check, then find its failure modes
After writing the initial check, systematically ask:
- False positive: what valid output would this check incorrectly score 1.0? (e.g., a regex for "1." also matches "10." or "1.5B")
- False negative: what invalid output would this check incorrectly score 0.0? (e.g., a check for "Category:" misses "## Market Category" as a heading)
- Malicious input: could a model learn to game this check trivially? (e.g., always appending "Sources: [1] [2] [3]" with no real links)
Fix the top 1–2 failure modes. Don't over-engineer — accept that code-based checks have limits, and cover the rest with LLM judges.
Step 4 — Add partial credit where it makes sense
Binary (0/1) scoring is fine when the property is binary (either you have 3 companies or you don't). Add partial credit when:
- There's a natural scale (0, 1, 2, 3 citations → 0.0, 0.33, 0.66, 1.0)
- You want to distinguish "completely missing" from "partially present"
Don't add partial credit just because it feels nicer — it complicates threshold-setting.
Step 5 — Test the evaluator on known examples before committing
Run the evaluator on at least:
- 1 clearly valid output (should score 1.0)
- 1 clearly invalid output (should score 0.0)
- 1 edge case that you're not sure about
If the evaluator scores an obviously valid output below 0.8, the check is too strict. If it scores an obviously invalid output above 0.2, the check is too loose.
Output Format
Write evaluators as a Python file with one function per evaluator. Each function:
- Has a clear docstring stating what it checks and what counts as pass/fail
- Returns a float or dict (not print statements)
- Is independently testable (no global state, no side effects)
def result_count(output: str, input: str, expected: str = None) -> dict:
"""
Checks that exactly N items appear in the output.
Pass: output contains exactly N numbered/ranked entries
Fail: output contains 0 to N-1 or N+ entries
Partial (0.5): output contains N-1 entries (close but not compliant)
Does NOT check whether the items are correct — use an LLM judge for content correctness.
"""
...
Common Patterns
Count with partial credit
import re
def result_count(output, input, expected=None):
"""
Checks that exactly N ranked/numbered items appear in the output.
Adapt the regex and expected count to your product's output format.
"""
matches = re.findall(r'(?:^|\n)\s*\d+[\.\)]\s+\S', output, re.MULTILINE)
expected_n = 3 # set to whatever your product should return
if len(matches) == expected_n:
return 1.0
if len(matches) == expected_n - 1:
return {"score": 0.5, "metadata": {"found": len(matches), "expected": expected_n}}
return {"score": 0.0, "metadata": {"found": len(matches), "expected": expected_n}}