跳至正文
OpenAgentSkill
浏览已开放 · 目录记录日期 2026-11-21 · 自动更新和账户服务暂时关闭。
搜索与研究

agentsop-domain-eval-set

Build and govern a 50-200 example domain-specific held-out benchmark sampled from real traffic. Distinct from public benchmarks (MMLU/HumanEval/GSM8K via lm-evaluation-harness) which measure GENERAL capability. Only a held-out domain set predicts whether THIS system works on YOUR data. Collect real examples, label, hold out (never train/prompt on it), size 50-200, version it, refresh on drift.

agentsope · 已有 AI 审核记录

安装与使用

运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。

npx skills add agentsope/SkillAlchemy --skill agentsop-domain-eval-set

技能说明


name: agentsop-domain-eval-set version: 0.1.0 phase: D tier: core frequency: high status: opinionated overlay: ENHANCE description: Build and govern a 50-200 example domain-specific held-out benchmark sampled from real traffic. Distinct from public benchmarks (MMLU/HumanEval/GSM8K via lm-evaluation-harness) which measure GENERAL capability. Only a held-out domain set predicts whether THIS system works on YOUR data. Collect real examples, label, hold out (never train/prompt on it), size 50-200, version it, refresh on drift.

domain-eval-set — Your Held-Out Domain Benchmark

"Compiled program beats baseline on a held-out test set (not the val set used in optimization)." — DSPy SOP exit criterion [dspy.ai/learn/optimization/overview/]

"Build the eval loop before optimizing anything. Every subsequent change must be gated on these numbers." — LlamaIndex SOP Stage 2

This is an ENHANCE overlay skill. It produces one artifact — a versioned, sealed, human-labeled set of 50–200 examples drawn from your domain — that other skills consume: [[agentsop-regression-gate]] enforces it on every PR, [[agentsop-metric-design]] defines the scoring function applied to each example, and [[lm-evaluation-harness]] runs the complementary public-capability axis. The core claim: public benchmarks tell you the model is smart in general; only a held-out domain set tells you it works on your task. The latter is the one that predicts production.


1. 何时激活 (When to Activate)

Activate when any of these is true:

  • "Does THIS system work on OUR data?" — someone is about to ship or trust an LLM/RAG/agent system and the only evidence is vibes, a demo, or a public benchmark number. You need a quantitative answer on the real distribution.
  • A public-benchmark number is being used as a deployment gate. Someone cites "92% on MMLU" or "passes HumanEval" to justify go-live. That measures general capability, not your task fit (AP-1). Force a domain set into the decision.
  • A model / prompt / retriever / chunking change needs a regression gate and no domain test set exists yet to gate against. You must build the set before [[agentsop-regression-gate]] can do its job.
  • Switching models (GPT-4o → a cheaper or local model). The public-bench gap may be small while the domain gap is large, or vice versa. Only your held-out set tells you which.
  • Production complaints don't match your eval scores. Either the set is stale (refresh, OP-DE06) or it never reflected the domain (rebuild from real traffic).

Do NOT activate for:

  • Pure capability comparison / academic reporting. "Which model is best at MMLU/GSM8K?" → that is [[lm-evaluation-harness]], not this skill.
  • One-off throwaway prototypes where no decision rides on quality and nothing ships. Don't build a benchmark for a script you'll delete tomorrow.
  • Tasks with an objective oracle already (compiler passes, exact DB match, schema validity gives ≥95% of signal) — the "eval set" is just running the oracle; you don't need curated held-out examples. Don't gold-plate.

2. 核心心智模型 (Core Mental Model)

"Public benchmarks measure general capability. A 50–200 example held-out domain set measures YOUR task. Only the latter predicts production."

Two orthogonal axes, constantly confused:

AxisWhat it measuresToolPredicts production?
General capabilityReasoning, knowledge, coding in general, on shared public tasks[[lm-evaluation-harness]] (MMLU, HumanEval, GSM8K, TruthfulQA)No — a proxy at best
Domain task fitWhether the system answers your users on your datathis skill (held-out domain set)Yes — this is the signal

A model can score 90% on MMLU and 40% on your insurance-claims triage. A model can score below SOTA on HumanEval and be perfect at your internal codebase's patterns. The public number and the domain number are nearly uncorrelated once you're past a basic capability floor. The public bench is a sanity check; the domain set is the decision.

Three corollaries (each maps to an SOP stage):

  1. Real beats synthetic. The set is sampled from real domain traffic (tickets, queries, logs, transactions), stratified, with edge cases pulled deliberately. Auto-generated QA pairs (LlamaIndex DatasetGenerator) are a fine bootstrap, but a model can ace generated questions and still fail real user phrasing. Generated sets do not replace a real held-out set (§7).

  2. Held out means SEALED. The held-out split is never shown to the optimizer, never pasted into a prompt as a few-shot demo, never used to pick chunk size or reranker, never in the fine-tune data. The moment it leaks, the number is inflated and meaningless (AP-2, OP-DE07). Per DSPy: the test set must be distinct from the val set used in optimization [dspy.ai/learn/optimization/overview/].

  3. Small but significant. 50–200 examples. Below ~30 you are "memorizing, not training" [dspy.ai/learn/optimization/overview/] and differences are noise. The set is small enough to label by hand and large enough to detect ~5–10pp regressions and to slice by segment.


3. SOP (Standard Operating Procedure)

0. Confirm activation (§1) — is the question "does this work on OUR data"?
1. COLLECT  — sample real domain examples; stratify; pull edge cases       (OP-DE01)
2. LABEL    — gold answer / reference / pass-fail; 2 annotators on subset  (OP-DE02)
3. HOLD OUT — split train/dev/test; SEAL the test split                    (OP-DE03)
4. SIZE     — land at 50-200; per-segment counts                           (OP-DE04)
5. VERSION  — hash + date + rubric; freeze as an artifact                  (OP-DE05)
6. LEAK-AUDIT — diff held-out vs demos / train / fine-tune data            (OP-DE07)
7. PAIR     — report alongside public bench; gate on the domain set        (OP-DE08)
   (later)
阅读完整来源 ↗