align-human
Use when the user has a judge/grader and human-labeled data, and wants to measure how well the judge agrees with humans, detect systematic biases, determine whether automatic evaluation can replace human review, or build a human-reduction roadmap. Also use when the user mentions calibration, TPR/TNR, judge validation, inter-rater agreement, Cohen's kappa, bias detection, or "is my automatic evaluation trustworthy." Merges the calibrate and align functions into one skill.
agentscope-ai · 已有 AI 审核记录
安装与使用
运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。
npx skills add agentscope-ai/OpenJudge --skill align-human技能说明
name: align-human description: > Use when the user has a judge/grader and human-labeled data, and wants to measure how well the judge agrees with humans, detect systematic biases, determine whether automatic evaluation can replace human review, or build a human-reduction roadmap. Also use when the user mentions calibration, TPR/TNR, judge validation, inter-rater agreement, Cohen's kappa, bias detection, or "is my automatic evaluation trustworthy." Merges the calibrate and align functions into one skill.
Align Human
Measure whether your automatic judge agrees with human judgment, detect where and why they disagree, and build a roadmap to reduce human review over time.
When to Activate
- You have a working judge/grader and 50+ human-labeled examples
- You want to know if the judge is trustworthy enough to replace human review
- You've noticed the judge's decisions being overturned by humans
- You're preparing to deploy an evaluation as a production gate
Checklist
You MUST create a task for each item and complete them in order:
- Load paired data — match judge verdicts with human labels
- Measure TPR/TNR — confusion matrix + per-stratum breakdown
- Calculate agreement — Cohen's kappa, Gwet's AC1, systematic bias
- Run bias detection — 5 systematic bias checks
- Analyze disagreements — cluster patterns + diagnose root causes
- Build human-reduction roadmap — 4-phase transition plan
- Confirm and record — one confirmation, then write results
Fast path: run the bundled script
Don't hand-write the calibration statistics — that is exactly where subtle bugs hide. Run
the bundled, tested script (scripts/calibration.py, standard library only, no OpenJudge
dependency):
python scripts/calibration.py --pairs pairs.jsonl # one paired file, OR
python scripts/calibration.py --verdicts verdicts.jsonl --labels labels.jsonl --stratum-key difficulty
Paired rows look like {"id","judge":"pass|fail","human":"pass|fail","stratum"?} (judge/human
may also be 1/0). It prints the confusion matrix, TPR/TNR/F1 with bootstrap 95% CIs, Cohen's
kappa, Gwet's AC1 (auto-flags the kappa paradox), directional bias, per-stratum TPR/TNR, and
the calibration gate verdict (calibrated / not_calibrated / insufficient_evidence;
exit code 0 only if calibrated). --json for machine output, --self-test to verify it.
Always report and interpret the actual numbers the script returns — TPR/TNR (with their 95% CIs), Cohen's kappa, Gwet's AC1, directional bias, per-stratum TPR/TNR, and the gate verdict — never just state that you ran it. If you don't yet have the paired verdicts/labels, say exactly what's missing (e.g. the judge verdicts file, or N more labels per class).
The Steps below explain what each number means and how to act on it — read them to interpret the script's output. The inline snippets are the reference behind the script; you normally just run the script rather than re-implementing it.
Step 1: Load and Pair Data
Human labels live in labels/<grader_name>.jsonl, one row per judged sample. Keep them
separate from the dataset so they can be re-paired with any judge run:
{"id": "sample_017", "label": "pass", # "pass"|"fail" (or 1/0); joins to a dataset row id
"annotator": "alice", "rationale": "Order number matches context.",
"timestamp": "2026-06-20T10:00:00Z", "schema_version": 1}
Match judge verdicts with human labels:
import json
# Load human labels and judge verdicts
labels = {item["id"]: item["label"] for item in json.load(open("labels.jsonl"))}
verdicts = json.load(open("runs/verdicts-dev.jsonl"))
# Pair them
paired = []
for v in verdicts:
if v["id"] in labels:
paired.append({
"id": v["id"],
"judge": v["verdict"], # "pass" or "fail"
"human": labels[v["id"]], # "pass" or "fail"
})
print(f"Paired: {len(paired)}, Unmatched: {len(verdicts) - len(paired)}")
# Warn if severe class imbalance
pass_rate = sum(1 for p in paired if p["human"] == "pass") / len(paired)
if pass_rate > 0.8 or pass_rate < 0.2:
print(f"WARNING: Human label pass rate is {pass_rate:.0%} — "
"kappa may be paradoxically low. Use Gwet's AC1 as complement.")
Step 2: Measure TPR/TNR
Build a confusion matrix and compute per-stratum metrics:
from openjudge.analyzer.validation import (
AccuracyAnalyzer, F1ScoreAnalyzer,
FalsePositiveAnalyzer, FalseNegativeAnalyzer,
)
# Convert to OpenJudge-compatible dataset with labels
analysis_dataset = [
{"query": p.get("query", ""), "response": p.get("response", ""),
"label": 1 if p["human"] == "pass" else 0}
for p in paired
]
# Binary grader results (1=pass, 0=fail)
grader_results = [
GraderScore(name="judge", score=1.0 if p["judge"] == "pass" else 0.0, reason="")
for p in paired
]
accuracy = AccuracyAnalyzer().analyze(analysis_dataset, grader_results, label_path="label")
f1 = F1ScoreAnalyzer().analyze(analysis_dataset, grader_results, label_path="label")
fpr = FalsePositiveAnalyzer().analyze(analysis_dataset, grader_results, label_path="label")
fnr = FalseNegativeAnalyzer().analyze(analysis_dataset, grader_results, label_path="label")
# TPR = 1 - FNR, TNR = 1 - FPR
tpr = 1 - fnr.false_negative_rate
tnr = 1 - fpr.false_positive_rate
print(f"TPR={tpr:.2f}, TNR={tnr:.2f}, F1={f1.f1_score:.2f}")
# Per-stratum breakdown if difficulty data exists
for stratum in ["easy", "boundary", "hard"]:
stratum_data = [d for d in analysis_dataset
if d.get("metadata", {}).get("difficulty") == stratum]
if len(str
阅读完整来源 ↗