跳至正文
OpenAgentSkill
浏览已开放 · 目录记录日期 2026-11-21 · 自动更新和账户服务暂时关闭。
AI 与知识库

meta-eval

Use when the user wants to build an evaluation system for an LLM/agent application but doesn't know where to start — they have traces, prompts, RAG pipelines, or nothing at all. Also use when the user mentions evaluation, eval, benchmarking, testing LLM quality, measuring agent performance, assessing RAG accuracy, or wants to compare prompts/models. This skill is the entry router: it asks diagnostic questions then recommends which sub-skill (local workflow) to use next.

agentscope-ai · 已有 AI 审核记录

安装与使用

运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。

npx skills add agentscope-ai/OpenJudge --skill meta-eval

技能说明


name: meta-eval description: > Use when the user wants to build an evaluation system for an LLM/agent application but doesn't know where to start — they have traces, prompts, RAG pipelines, or nothing at all. Also use when the user mentions evaluation, eval, benchmarking, testing LLM quality, measuring agent performance, assessing RAG accuracy, or wants to compare prompts/models. This skill is the entry router: it asks diagnostic questions then recommends which sub-skill (local workflow) to use next.

Meta Eval

Entry router for the eval skill collection. You diagnose what the user has and route them to the right sub-skill. You don't do evaluation yourself — you're the triage desk.

Each sub-skill is self-contained: it carries inline the data shapes, statistics, and data principles it needs, so it can be installed and used on its own.

Checklist

You MUST create a task for each item and complete them in order:

  1. Ask 4 diagnostic questions — data, labels, stakes, domain knowledge
  2. Match triage table — map user scenario to sub-skill
  3. Recommend sub-skill — tell the user which workflow to use and why
  4. Record routing decision — write a brief summary of what was diagnosed and recommended

Diagnostic Questions

Ask these 4 questions (all at once — don't drip-feed):

To route you to the right evaluation skill, I need to understand your situation:

1. What data do you have?
   a) Agent traces / production logs
   b) Product spec / design docs
   c) Nothing yet — starting from scratch

2. Do you have human labels?
   a) Yes, ≥50 labeled examples
   b) Some, but fewer than 50
   c) None

3. What are the stakes?
   a) Low — internal experimentation, exploring options
   b) Production — customer-facing, quality matters
   c) Regulated — compliance requirements, audit trail needed

4. How well do you know this evaluation domain?
   a) Very well — have clear standards and criteria
   b) Somewhat — general idea but need structure
   c) Not well — exploring what "good" even means

Shortcut rule: data_form + label_status already determine the entry workflow (see triage table). The moment those two are clear — even if stakes and domain knowledge are not — give the provisional recommendation AND ask the remaining questions in the same message. stakes and user_prior refine the downstream path (how much calibration rigor, how fast a path), not the entry point. Never make the user wait a round-trip for a route you can already determine.

Example: "no logs, no labels" → recommend 08-bootstrap now, and ask stakes/domain to tune the roadmap. Don't reply with only the questionnaire.

Triage Table

Match the user's situation to a sub-skill:

These are local workflows under skills/eval_pipeline/, not packages to install — "use" a workflow means open and follow that sub-skill.

User says / hasUse workflowWhat it does
"I have agent traces / production logs"01-eval-designExtract eval dimensions from traces → design dataset in OpenJudge format
"I have principles/criteria but need test data"01-eval-designStratified sampling + adversarial generation → OpenJudge dataset
"I have principles but don't know which graders to use"02-metric-designSelect OpenJudge graders by output type → generate executable pipeline code
"I changed my prompt, is it better?"06-prompt-regressionA/B comparison with PairwiseAnalyzer, win rates + statistical significance
"I have a RAG system"05-rag-evalRetrieval + generation separation, hallucination detection, diagnostic matrix
"I have a judge + labels, want to check accuracy"03-align-humanTPR/TNR calibration, kappa agreement, human-reduction roadmap
"I want to do safety/security testing"07-redteamAttack surface analysis, jailbreak/injection generation, harmfulness grading
"I've run multiple skills, want a comprehensive report"04-eval-reportCross-skill analysis, maturity dashboard, prioritized actions
"Nothing — starting from scratch"08-bootstrapZero-shot grader generation via SimpleRubricsGenerator, v0 in 30 minutes
None of the above match—Say "this scenario isn't covered yet" and suggest filing an issue

Output

After diagnosis, respond with:

Diagnosis: data=[data_form] | labels=[label_status] | stakes=[value or "asking"] | domain=[value or "asking"]

Recommended workflow: `[skill-name]`   (provisional if stakes/domain unknown)

Why: [one sentence explaining the routing decision from data_form + label_status]

What this workflow will do: [one sentence about the output — e.g., "produces an
OpenJudge-compatible dataset with stratified sampling"]

To refine the path, also tell me: [stakes / domain knowledge, if still unknown]

Recommend exactly ONE workflow as the immediate next step. Do NOT list a second workflow as a current action — that splits the user's focus. If they ask "what comes after," point them to the Canonical Workflow below as a map for later, explicitly framed as "once you finish [recommended workflow]," not as a second thing to do now.

A ? marks a field you are still asking about. Give the recommendation now; refine later.

Canonical Workflow (the standard lifecycle)

Most evaluation builds follow this order. Use it to sequence sub-skills and to state preconditions — recommend the next workflow only when its inputs exist.

1. 00-meta-eval        route to the right entry workflow
2. entry point:
     - have traces/spec  → 01-eval-design   (
阅读完整来源 ↗