跳至正文
OpenAgentSkill
浏览已开放 · 目录记录日期 2026-11-21 · 自动更新和账户服务暂时关闭。
AI 与知识库

eval-llm-judge

Write LLM-as-judge evaluator prompts for AI products. Use this skill whenever you need to evaluate semantic properties of AI outputs that cannot be checked programmatically — output quality, tone, factual accuracy, edge case handling, reference alignment. Also use when auditing existing judge prompts for bias, vagueness, or compound criteria. Output is a judge prompt string and a Python runner function, saved to the project's evaluators file.

Calibre-Labs · 已有静态检查记录

安装与使用

运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。

npx skills add Calibre-Labs/reforge-ai-evals --skill eval-llm-judge

技能说明


name: eval-llm-judge description: > Write LLM-as-judge evaluator prompts for AI products. Use this skill whenever you need to evaluate semantic properties of AI outputs that cannot be checked programmatically — output quality, tone, factual accuracy, edge case handling, reference alignment. Also use when auditing existing judge prompts for bias, vagueness, or compound criteria. Output is a judge prompt string and a Python runner function, saved to the project's evaluators file.

LLM Judge Evaluator Skill

An LLM-as-judge evaluator uses a language model to score another model's output against a defined criterion. It handles semantic properties that code-based checks cannot: is the ranking defensible? Did the agent handle ambiguity gracefully? Does the output align with the reference?

The tradeoff: judges are slower, more expensive, and non-deterministic. They require calibration against human labels before you can trust their scores. Use them only after code-based checks have covered the structural requirements.

When to use this skill

Write an LLM judge when the property requires reasoning that a human reviewer would apply:

  • Output correctness: are the results correct and in a defensible order or format?
  • Edge case handling: did the agent appropriately flag ambiguity or decline gracefully?
  • Reference alignment: does the output agree with a gold-standard expected response?
  • Tone and framing: is the hedging appropriate for a speculative or high-stakes claim?
  • Faithfulness: does the output stay within what the cited sources actually say?

Do NOT write an LLM judge for:

  • Properties a code check can handle (count, format, presence) — these are cheaper and faster
  • Compound criteria — write one judge per criterion, not one judge that checks everything

Workflow

Step 1 — Define exactly ONE criterion

The most common failure in judge design is compound criteria: "Is the output accurate, well-formatted, and appropriately hedged?" This produces judges that are hard to calibrate and whose scores are uninterpretable — a 0 could mean any of three things.

Write the criterion as one sentence that completes: "A good output [verb phrase]."

Good: "A good output ranks the top 3 companies in defensible order based on the stated metrics." Good: "A good output explicitly states its interpretation when the query is ambiguous." Bad: "A good output is accurate, well-cited, and appropriately hedged." (three criteria)

If you're tempted to write a compound criterion, split it into two judges.

Step 2 — Write binary pass/fail definitions

Binary (PASS/FAIL) forces the judge to commit and produces clean metrics.

Do not use a 1–5 Likert scale. It collapses to 3s and 4s and tells you nothing. Do not add a BORDERLINE tier — it becomes a crutch and makes TPR/TNR harder to interpret.

Write definitions for each level that are:

  • Specific: reference the criterion, not generic quality
  • Mutually exclusive: a judge should not reasonably assign two levels to the same output
  • Grounded: name what counts as evidence, not just "good" vs "bad"

Step 3 — Write 3 few-shot examples

The examples are the most important part of the prompt. They calibrate the judge's interpretation of the criterion and tell it what evidence to look for.

Required: one clear PASS, two clear FAILs covering different failure modes.

Rules for good examples:

  1. Use two distinct FAIL examples. Different failure modes teach the judge where the line is better than a single example. Spend more time on these than on the PASS.
  2. Use real-looking outputs, not toy ones. The output in the example should look like what your model actually produces — same formatting, similar length, same hedging patterns.
  3. Write the critique before the score. The critique teaches the judge why to assign a score, not just what score to assign. The judge should be able to derive the score from the critique.
  4. Don't use inputs from your eval dataset. Example inputs and outputs should be entirely distinct from the queries in your test set, to avoid contaminating judge behavior.
  5. Match the domain. Use examples from your actual product — same output format, same hedging patterns, same vocabulary. Don't use generic QA examples; the judge will pattern-match to the wrong signals.

Step 4 — Require structured output

The judge's output should be a JSON object, with the critique before the score:

{"critique": "2-3 sentence explanation of why this output passes/fails", "score": "PASS"}

Critique-first ordering matters: it forces the judge to reason before committing to a score, which produces more reliable results than score-first or score-only formats.

Step 5 — Validate the judge against human labels

Before using a judge in production evals, validate it:

  1. Label 20–40 outputs by hand (10–20 PASS, 10–20 FAIL)
  2. Run the judge on the same outputs
  3. Calculate TPR (when human says PASS, judge also says PASS) and TNR (when human says FAIL, judge also says FAIL)
  4. Target: TPR > 90% and TNR > 90%. Minimum viable: both > 80%
  5. If failing, read the mismatches — they reveal criterion ambiguity or missing example coverage

If a judge consistently fails on a specific sub-type of output, add a few-shot example covering that sub-type. If the criterion itself is ambiguous, rewrite it before adding more examples.


Prompt Template

JUDGE_PROMPT = """\
You are evaluating [AI product description].

## Evaluation Criterion
**[Criterion Name]**: [One-sentence definition of what a good output does.]

## Scoring
- **PASS**: [Specific description of what qualifies as a pass, grounded in the criterion.]
- **FAIL**: [Specific description of what qualifies as a fail.]

## Few-Shot Examples

### Example 1 — PASS
Input: "[example input]"
Output excerpt: "[example output]"
Result: {{"critique": "[why this passes]", "score": "
阅读完整来源 ↗