llm-align
Analyze alignment between LLM judge scores and human labels in an eval dataset. Use this skill whenever someone wants to evaluate how well an LLM judge agrees with human reviewers, calculate TPR/TNR, investigate disagreements, or improve a scorer prompt. Trigger on phrases like: "calculate TPR/TNR", "judge alignment", "where does the LLM disagree with humans", "improve my scorer", "why is the LLM getting these wrong", "compare human labels to LLM scores", "evaluate my eval", or any time someone shares eval results and asks about score quality or judge calibration.
Calibre-Labs · 已有静态检查记录
安装与使用
运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。
npx skills add Calibre-Labs/reforge-ai-evals --skill llm-align技能说明
name: llm-align description: > Analyze alignment between LLM judge scores and human labels in an eval dataset. Use this skill whenever someone wants to evaluate how well an LLM judge agrees with human reviewers, calculate TPR/TNR, investigate disagreements, or improve a scorer prompt. Trigger on phrases like: "calculate TPR/TNR", "judge alignment", "where does the LLM disagree with humans", "improve my scorer", "why is the LLM getting these wrong", "compare human labels to LLM scores", "evaluate my eval", or any time someone shares eval results and asks about score quality or judge calibration.
LLM Judge Alignment Analysis
You are helping analyze how well an LLM-based judge aligns with human labels in an eval dataset. Work through the five phases below in order, confirming with the user at each ambiguous step before proceeding.
The methodology works regardless of eval platform (Braintrust, LangSmith, custom CSV/JSONL, or any other format). Phase 1 covers how to get the data you need from different sources.
Phase 1: Discovery
Goal: Figure out what score fields exist and which are human labels vs. LLM judge scores.
Step 1 — Get the eval results
Ask the user how their eval results are stored:
Option A — Local file (CSV or JSONL)
Read the file directly. Look for columns or fields that contain numeric or categorical scores.
Common patterns: human_score, llm_score, <judge_name>_score, label, rating.
Option B — Eval platform (Braintrust, LangSmith, etc.) Ask for the experiment/run ID or URL. Use the platform's API or MCP tools to fetch rows. You want: each row's input, the human label field, and the LLM judge score field.
Typical Braintrust query (adapt syntax to other platforms):
select: id, span_id, scores
object_type: experiment
object_ids: ["<experiment-id>"]
where: is_root = true
limit: 500
Option C — The user pastes results directly Accept a JSON array, CSV snippet, or any structured format. Parse it in-context.
Step 2 — Identify human vs. LLM score fields
From the data, extract all score fields. For each:
- What are the distinct values? (binary 0/1, categorical PASS/FAIL, continuous 0–1)
- How many rows have this field populated?
- Does the name suggest human or LLM origin? (e.g.,
human_*,ground_truth,labelare likely human; criterion-named fields are likely LLM)
Present your findings and ask the user to confirm: "Which field is the human label and which is the LLM judge score? I'll skip rows where either is missing — is that OK?"
Technical notes (platform-specific)
- Braintrust: hyphenated field names need bracket notation:
scores["metric-scope"]notscores.metric-scope. Root spans are where scores aggregate — filteris_root = true. Theidfield ≠span_id; select both for later trace lookups. - CSV/JSONL: parse scores as floats; watch for string "PASS"/"FAIL" that need mapping to 1.0/0.0.
Phase 2: Confusion Matrix + TPR/TNR
Goal: Compute the full confusion matrix and key classification metrics.
Setup
- Positive (1) = the human label value the user defines as "pass" (default: 1 or "PASS")
- Negative (0) = the other value (default: 0 or "FAIL")
- Threshold: for continuous LLM scores (0.0–1.0), default to ≥ 0.5 = predicted positive. Ask if the score looks continuous.
Compute the confusion matrix
Group rows into four cells:
- TP: human = positive, LLM = positive
- FN: human = positive, LLM = negative (judge misses a real pass)
- FP: human = negative, LLM = positive (judge passes something that should fail)
- TN: human = negative, LLM = negative
Report
| LLM: Positive | LLM: Negative | |
|---|---|---|
| Human: Positive | TP = N | FN = N |
| Human: Negative | FP = N | TN = N |
Then:
- TPR (sensitivity/recall) = TP / (TP + FN) — how well the LLM catches true positives
- TNR (specificity) = TN / (TN + FP) — how well the LLM rejects true negatives
- Total rows used, rows skipped (missing scores)
Interpret the numbers — don't just show the math. E.g.: "TPR of 85% means the judge misses 15% of cases your human reviewers would pass."
Phase 3: Disagreement Investigation
Goal: Understand why the LLM got each disagreement wrong by reading the scorer's reasoning.
Step 1 — Pull disagreement rows
Filter to rows where human label ≠ LLM score (accounting for threshold if continuous). Show up to 20 disagreements — enough to find patterns without overwhelming.
For each row, gather:
- The input
- The AI output (if available)
- Human verdict vs. LLM verdict
- The LLM's reasoning (from the scorer's trace/metadata if available)
Step 2 — Retrieve scorer reasoning
If your eval platform stores scorer reasoning (a critique or rationale alongside the score):
- Braintrust: fetch the scorer span using
root_span_id = '<span_id>'and look foroutput.metadata.rationaleor similar - LangSmith: check the run's feedback or evaluator output fields
- Custom: look for a
critique,rationale, orreasonfield in the eval output
If no reasoning is stored, re-run the judge on the disagreement rows locally and capture the reasoning output.
Step 3 — Summarize patterns
Group errors into patterns — don't list them one by one. Common patterns:
- Too strict FNs: LLM fails responses for a pedantic reason humans overlook
- Too lenient FPs: LLM passes responses that violate the spirit of the criterion
- Ambiguous criteria: the rubric has a gap the LLM fills differently than humans would
Phase 4: Prompt Fix Suggestions
Goal: Suggest targeted, surgical edits to the scorer prompt — not a full rewrite.
Based on the disagreement patterns from Phase 3:
- Quote the specific part of the existing scorer prompt that's causing the issue
- Propose the minimal change — a single sentence addition, a clarification to an example, a new edge case in the rubric
- Explain ho