eval-llm-judge
Write LLM-as-judge evaluator prompts for AI products. Use this skill whenever you need to evaluate semantic properties of AI outputs that cannot be checked programmatically — output quality, tone, factual accuracy, edge case handling, reference alignment. Also use when auditing existing judge prompts for bias, vagueness, or compound criteria. Output is a judge prompt string and a Python runner function, saved to the project's evaluators file.
Calibre-Labs · 已有静态检查记录
安装与使用
运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。
npx skills add Calibre-Labs/reforge-ai-evals --skill eval-llm-judge技能说明
name: eval-llm-judge description: > Write LLM-as-judge evaluator prompts for AI products. Use this skill whenever you need to evaluate semantic properties of AI outputs that cannot be checked programmatically — output quality, tone, factual accuracy, edge case handling, reference alignment. Also use when auditing existing judge prompts for bias, vagueness, or compound criteria. Output is a judge prompt string and a Python runner function, saved to the project's evaluators file.
LLM Judge Evaluator Skill
An LLM-as-judge evaluator uses a language model to score another model's output against a defined criterion. It handles semantic properties that code-based checks cannot: is the ranking defensible? Did the agent handle ambiguity gracefully? Does the output align with the reference?
The tradeoff: judges are slower, more expensive, and non-deterministic. They require calibration against human labels before you can trust their scores. Use them only after code-based checks have covered the structural requirements.
When to use this skill
Write an LLM judge when the property requires reasoning that a human reviewer would apply:
- Output correctness: are the results correct and in a defensible order or format?
- Edge case handling: did the agent appropriately flag ambiguity or decline gracefully?
- Reference alignment: does the output agree with a gold-standard expected response?
- Tone and framing: is the hedging appropriate for a speculative or high-stakes claim?
- Faithfulness: does the output stay within what the cited sources actually say?
Do NOT write an LLM judge for:
- Properties a code check can handle (count, format, presence) — these are cheaper and faster
- Compound criteria — write one judge per criterion, not one judge that checks everything
Workflow
Step 1 — Define exactly ONE criterion
The most common failure in judge design is compound criteria: "Is the output accurate, well-formatted, and appropriately hedged?" This produces judges that are hard to calibrate and whose scores are uninterpretable — a 0 could mean any of three things.
Write the criterion as one sentence that completes: "A good output [verb phrase]."
Good: "A good output ranks the top 3 companies in defensible order based on the stated metrics." Good: "A good output explicitly states its interpretation when the query is ambiguous." Bad: "A good output is accurate, well-cited, and appropriately hedged." (three criteria)
If you're tempted to write a compound criterion, split it into two judges.
Step 2 — Write binary pass/fail definitions
Binary (PASS/FAIL) forces the judge to commit and produces clean metrics.
Do not use a 1–5 Likert scale. It collapses to 3s and 4s and tells you nothing. Do not add a BORDERLINE tier — it becomes a crutch and makes TPR/TNR harder to interpret.
Write definitions for each level that are:
- Specific: reference the criterion, not generic quality
- Mutually exclusive: a judge should not reasonably assign two levels to the same output
- Grounded: name what counts as evidence, not just "good" vs "bad"
Step 3 — Write 3 few-shot examples
The examples are the most important part of the prompt. They calibrate the judge's interpretation of the criterion and tell it what evidence to look for.
Required: one clear PASS, two clear FAILs covering different failure modes.
Rules for good examples:
- Use two distinct FAIL examples. Different failure modes teach the judge where the line is better than a single example. Spend more time on these than on the PASS.
- Use real-looking outputs, not toy ones. The output in the example should look like what your model actually produces — same formatting, similar length, same hedging patterns.
- Write the critique before the score. The critique teaches the judge why to assign a score, not just what score to assign. The judge should be able to derive the score from the critique.
- Don't use inputs from your eval dataset. Example inputs and outputs should be entirely distinct from the queries in your test set, to avoid contaminating judge behavior.
- Match the domain. Use examples from your actual product — same output format, same hedging patterns, same vocabulary. Don't use generic QA examples; the judge will pattern-match to the wrong signals.
Step 4 — Require structured output
The judge's output should be a JSON object, with the critique before the score:
{"critique": "2-3 sentence explanation of why this output passes/fails", "score": "PASS"}
Critique-first ordering matters: it forces the judge to reason before committing to a score, which produces more reliable results than score-first or score-only formats.
Step 5 — Validate the judge against human labels
Before using a judge in production evals, validate it:
- Label 20–40 outputs by hand (10–20 PASS, 10–20 FAIL)
- Run the judge on the same outputs
- Calculate TPR (when human says PASS, judge also says PASS) and TNR (when human says FAIL, judge also says FAIL)
- Target: TPR > 90% and TNR > 90%. Minimum viable: both > 80%
- If failing, read the mismatches — they reveal criterion ambiguity or missing example coverage
If a judge consistently fails on a specific sub-type of output, add a few-shot example covering that sub-type. If the criterion itself is ambiguous, rewrite it before adding more examples.
Prompt Template
JUDGE_PROMPT = """\
You are evaluating [AI product description].
## Evaluation Criterion
**[Criterion Name]**: [One-sentence definition of what a good output does.]
## Scoring
- **PASS**: [Specific description of what qualifies as a pass, grounded in the criterion.]
- **FAIL**: [Specific description of what qualifies as a fail.]
## Few-Shot Examples
### Example 1 — PASS
Input: "[example input]"
Output excerpt: "[example output]"
Result: {{"critique": "[why this passes]", "score": "
阅读完整来源 ↗