跳至正文
OpenAgentSkill
浏览已开放 · 目录记录日期 2026-11-21 · 自动更新和账户服务暂时关闭。
设计与 UI

metric-design

Use when the user has evaluation principles or a dataset but needs help choosing the right graders, designing evaluation metrics, creating LLM-as-judge prompts, combining multiple metrics into a composite score, or building an automated evaluation pipeline. Also use when the user mentions grader selection, metric design, judge prompt engineering, rubric design, evaluation pipeline code, or "how to evaluate [X] automatically." Outputs executable OpenJudge pipeline code.

agentscope-ai · 已有 AI 审核记录

安装与使用

运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。

npx skills add agentscope-ai/OpenJudge --skill metric-design

技能说明


name: metric-design description: > Use when the user has evaluation principles or a dataset but needs help choosing the right graders, designing evaluation metrics, creating LLM-as-judge prompts, combining multiple metrics into a composite score, or building an automated evaluation pipeline. Also use when the user mentions grader selection, metric design, judge prompt engineering, rubric design, evaluation pipeline code, or "how to evaluate [X] automatically." Outputs executable OpenJudge pipeline code.

Metric Design

Select, configure, and combine evaluation graders into a working pipeline. You choose the right tool for each evaluation dimension — from zero-cost code checks to LLM judges — and produce executable GradingRunner code that runs on OpenJudge.

Requires OpenJudge (pip install py-openjudge). This skill is intentionally SDK-centric — grader selection, GradingRunner, and aggregators are OpenJudge APIs. The design/decision logic still applies if you use another harness; only the code does not.

When to Activate

  • User has eval dimensions/principles but doesn't know which grader type to use
  • User wants to write an LLM-as-judge prompt for a specific failure mode
  • User needs a composite score combining multiple evaluation dimensions
  • User wants to auto-generate graders from labeled data instead of writing them manually
  • User's current evaluation is all LLM-based and too expensive/too slow

Checklist

You MUST create a task for each item and complete them in order:

  1. Select grader types — per dimension, pick the right grader class
  2. Create custom graders — write judge prompts (4-component) or function graders
  3. Auto-generate if applicable — use OpenJudge Generator for cold starts
  4. Run anti-pattern scan — check for Likert, missing few-shot, vague criteria
  5. Build pipeline code — assemble GradingRunner with graders + aggregators

Step 1: Select Grader Type Per Dimension

For each evaluation dimension, walk this decision tree (first match wins):

1. Can a deterministic rule check this?
   → StringMatchGrader / JsonValidatorGrader / FunctionGrader (zero cost, 100% consistent)
   Examples: exact match for classification labels, regex for format checks,
             JSON schema validation, keyword presence/absence

2. Does it require semantic understanding of text quality?
   → LLMGrader with built-in class (low cost, pre-optimized)
   Examples: CorrectnessGrader (factual match), RelevanceGrader (on-topic check),
             HallucinationGrader (faithfulness to context)

3. Does it involve agent behavior (tool calls, planning, memory)?
   → Agent-specific LLMGrader
   Examples: ToolSelectionGrader, TrajectoryAccuracyGrader, MemoryAccuracyGrader

4. Does it involve code execution or syntax?
   → CodeExecutionGrader / SyntaxCheckGrader
   Examples: test case pass rate, syntax validity, code style checks

5. Does it require external tool calls to verify (web search, database lookup)?
   → AgenticGrader (expensive, use only when necessary)
   Examples: fact-checking against live sources, cross-referencing databases

Grader Selection Cheat Sheet

Output typeRecommended graderCost
Classification labelStringMatchGraderFree
JSON structureJsonValidatorGrader + JsonMatchGraderFree
Free text correctnessCorrectnessGraderLLM call
Factual accuracy (grounded)HallucinationGraderLLM call
Response relevanceRelevanceGraderLLM call
Instruction followingInstructionFollowingGraderLLM call
Tool call selectionToolSelectionGraderLLM call
Agent trajectoryTrajectoryAccuracyGraderLLM call
Code correctnessCodeExecutionGraderFree
Custom quality checkCustom LLMGraderLLM call
External fact verificationAgenticGraderLLM + tool calls

Why this order matters: Every LLM-based grader adds cost, latency, and non-determinism. A StringMatchGrader costs nothing and always gives the same answer. Exhaust deterministic options before reaching for an LLM judge.

Step 2: Create Custom Graders

LLMGrader: The Four-Component Template

When no built-in grader fits, create a custom LLMGrader. Every judge prompt needs exactly these four components (adapted from community best practice):

Component 1 — Task & Criterion: What this judge evaluates. One thing only.

You are evaluating whether a customer support response correctly identifies
and uses the customer's order number from the conversation context.

Component 2 — Binary Pass/Fail Definitions: Concrete, observable conditions.

PASS: The response references the correct order number exactly as it appears
in the context. If multiple orders exist, the response addresses the right one.

FAIL: The response uses a wrong order number, omits the order number when one
was provided, or references an order not present in the context.

Why binary and not Likert? Because two human annotators agree on "pass vs fail" far more often than on "3 vs 4 out of 5." Binary forces a clear decision boundary. If you need severity levels, use multiple binary judges (e.g., "factually wrong" + "dangerously wrong").

Component 3 — Few-Shot Examples: At minimum 1 pass, 1 fail, 1 borderline. The borderline example is the most valuable — it teaches the judge where the boundary is.

Example 1 (PASS):
Context: "Order #12345: shipped May 10"
Response: "Your order #12345 was shipped on May 10 and arrives May 12."
Critique: The response uses the exact order number (#12345) and matches the
ship date from context. No fabrication or omission.
Result: Pass

Example 2 (FAIL):
Context: "Order #12345: shipped May 10"
Response: "Your order #12346 is on its way!"
Critique: The response uses order #12346 but the context only mentions #12345.
This is a fabricated order number, not a typo — #12346 doesn't exist.
Resu
阅读完整来源 ↗