跳至正文
OpenAgentSkill
浏览已开放 · 目录记录日期 2026-11-21 · 自动更新和账户服务暂时关闭。
其他技能

Skill Eval Harness

Agent Skill evaluation harness for paired variants, trace artifacts, and runner adapters

adewale · 已有 AI 审核记录

安装与使用

运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。

npx skills add adewale/skill-eval-harness

技能说明

Skill Eval Harness

Skill Eval Harness is a Python CLI that measures the causal lift of an Agent Skill: it runs the same case, model, and repetition with and without the skill, validates that exact experimental identity, then reports what changed, what passed, and whether the eval leaked its own answer. It reads evals/shared-benchmark.json, emits answer-key-safe task rows, grades files under eval-runs/ locally and deterministically — no model call in the grade path — and writes benchmark reports you can diff across variants.

General eval frameworks (openai/evals, vitest-evals, viteval) score one output against a rubric. This one measures the difference the skill makes, and spends its surface area on keeping that difference honest: paired with/without comparison, tune/holdout/holdback split discipline, leakage lint, materialized ablations with provenance gates, and per-model lift. None of those frameworks have them, and they are what make a reported number trustworthy rather than merely green.

Questions this helps answer

QuestionCommand/report to use
Does this skill improve outputs compared with no skill at all?prepare paired with_skill / without_skill rows, then benchmark paired lift and significance.
Which prompts improved, regressed, saturated, or showed no lift?benchmark case_flags, render-viewer, and error-analysis.
Is the skill worth its extra tokens or dollars?profile-skill, token-overhead, cost-summary, and lift-per-dollar summaries.
Did my latest skill edit introduce a regression?Re-run the same manifest, inspect ablation_regressions, trend, and render-viewer --previous-workspace.
Which instruction, checklist, reference, scr