Skip to content
OpenAgentSkill
Browsing is open · Catalog captured 2026-11-21 · Automatic updates and account services are paused.
Security

eval-report

Use when the user has run multiple evaluation skills and wants a comprehensive analysis — maturity assessment, cross-skill signals, trends, prioritized actions, and an executive summary. Also use when the user mentions eval health check, evaluation audit, ship readiness, evaluation maturity, or "how good is my evaluation system itself." This is a read-only analysis skill.

agentscope-ai · AI review on file

Installation

Read the source and dependencies before running a command. Copying does not install or execute anything.

npx skills add agentscope-ai/OpenJudge --skill eval-report

About this skill


name: eval-report description: > Use when the user has run multiple evaluation skills and wants a comprehensive analysis — maturity assessment, cross-skill signals, trends, prioritized actions, and an executive summary. Also use when the user mentions eval health check, evaluation audit, ship readiness, evaluation maturity, or "how good is my evaluation system itself." This is a read-only analysis skill.

Eval Report

Synthesize everything from your evaluation journey into a comprehensive report. This skill is read-only — it analyzes what exists, doesn't create new graders or datasets.

When to Activate

  • You've run 2+ evaluation skills and want the big picture
  • You need to report evaluation status to non-technical stakeholders
  • You're making a ship/no-ship decision and need evidence
  • The evaluation system has been running for a while — time for a health check

Checklist

You MUST create a task for each item and complete them in order:

  1. Inventory scan — catalog everything in eval-design.md + runs/ history
  2. Maturity assessment — 5 dimensions × 4 levels
  3. Cross-skill signal synthesis — consistent findings + contradictions
  4. Weakness diagnosis — failure concentration, correlations, stratum gaps
  5. Root cause classification — system / metric / data / unclear
  6. Prioritized recommendations — P0/P1/P2 actions with impact estimates
  7. Executive summary — ship readiness + top 3 risks + next actions

Step 1: Inventory Scan

Read eval-design.md and all runs/ directories. Build a timeline:

Timeline:
  2026-04-15  01-eval-design   → 5 failure modes → 3 dimensions from 200 traces
  2026-04-18  02-metric-design → 4 graders configured (2 LLM + 1 rule + 1 executable)
  2026-04-25  (evaluation run) → 90-sample stratified dataset scored
  2026-05-01  03-align-human   → 2 judges Phase 3, 1 Phase 2, 1 Phase 1 (TPR/TNR + kappa)
  2026-05-10  07-redteam       → safety audit not yet run

Report key metrics:

  • Total skills run, total principles, total labels
  • Calibrated judges: X of Y (with TPR/TNR range)
  • Last activity date per skill

Step 2: Maturity Assessment

Rate the evaluation system across 5 dimensions:

DimensionL1 (Initial)L2 (Developing)L3 (Established)L4 (Optimizing)
Failure DiscoveryNo systematic analysisFailure modes identifiedCoverage validated with stratificationContinuous triage from production
Judge Qualityv0 uncalibrated onlySome calibrated (TPR/TNR measured)All calibrated with CICalibrated + aligned with humans
Label Coverage< 50 labels50-200 labels200+ stratified labelsCoverage audit passed, drift monitored
Safety CoverageNo redteamingAd-hoc redteam runSystematic redteam with policy docContinuous redteam with sign-off
Human AlignmentNo alignment dataKappa measured for some judgesKappa ≥ 0.8 for all judgesHuman spot-check only, quarterly audit

Scoring rule: The overall maturity level is the minimum across dimensions (weakest link principle). If 4 dimensions are L3 but Safety is L1, the system is L1.

Step 3: Cross-Skill Signal Synthesis

Consistent Signals (high confidence)

Find themes confirmed by multiple skills. Example:

  • "Factuality is the top risk" — evidence chain:
    • 01-eval-design: #1 failure mode (38% prevalence in traces)
    • 02-metric-design: weighted as the highest-impact dimension
    • 03-align-human: TPR=0.92 TNR=0.88 (confirmed measurable)

Contradictions (needs investigation)

Find where skills disagree. These are the most valuable findings:

  • "02-metric-design weighted hallucination as a top signal, but 03-align-human shows the hallucination judge has TPR=0.74" → Possible explanations: the judge prompt captures surface patterns, not real hallucination. Or the judge prompt needs refinement, or the labels are noisy.

Coverage Gaps (blind spots)

What hasn't been touched by any skill?

  • "01-eval-design coverage shows multilingual input_type n=0, no workflow has addressed non-English queries"

Step 4: Weakness Diagnosis

Failure Concentration

Which principle/grader has the lowest pass rate? Where are failures clustering?

Failure Correlation

Compute Jaccard similarity between principle pairs — when sample A fails on principle X, does it also fail on principle Y? Highly correlated pairs (Jaccard > 0.5) likely share a root cause.

Per-Stratum Weakness

Which difficulty stratum performs worst across all principles? If boundary stratum TPR < 0.7 for 3 of 4 principles, boundary discrimination is a systemic weakness.

Step 5: Root Cause Classification

For each weakness area, classify the root cause:

TypeDefinitionKey indicator
system_problemThe application itself performs poorlyLow pass rate + high judge-human agreement
metric_problemThe judge/eval is flawedLow pass rate + low judge-human agreement
data_problemThe eval dataset isn't representative01-eval-design coverage shows thin strata OR label drift detected
unclearNot enough evidenceConflicting signals, need more data

This classification is critical — fixing a metric problem by changing the system (or vice versa) wastes effort.

Step 6: Prioritized Recommendations

Generate P0/P1/P2 actions. Each must include: priority, concrete action, current state, target state, expected impact, and the skill to use.

🔴 P0 | Calibrate hallucination judge
       Current: TPR=0.74 (below 0.8 threshold)
       Target:  TPR >= 0.8
       Impact:  Judge becomes
Read the complete source ↗