跳至正文
OpenAgentSkill
浏览已开放 · 目录记录日期 2026-11-21 · 自动更新和账户服务暂时关闭。
浏览器与自动化

eval-measure

Use when the user asks if an AI app is measured correctly, needs Galileo metrics, expected-output contracts, metric profiles, eval gates, or measurement before optimizing.

Galileo-Agent-Labs · 已有静态检查记录

安装与使用

运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。

npx skills add Galileo-Agent-Labs/eval-engineer --skill eval-measure

技能说明


name: eval-measure description: Use when the user asks if an AI app is measured correctly, needs Galileo metrics, expected-output contracts, metric profiles, eval gates, or measurement before optimizing.

Eval Measure

Use this skill before optimization or broad fixture work. Its job is to make the measurement contract explicit.

Required Reference

Use skills/eval-engineer/references/metric-profile-checklist.md and skills/eval-engineer/assets/metric-profile-template.md.

Do

  • Define risk profile and quality dimensions.
  • Write the full expected-output contract: expected decision, required and forbidden citations, tools, answer constraints, abstention, permissions, and safety requirements.
  • Include retrieved-source gates when source authority matters: required_retrieved_sources, forbidden_retrieved_sources, and whether final citations alone are sufficient for the case risk.
  • Prefer independent observations over agent self-reports for safety gates. Check answer text, citations, retrieved source IDs, tool calls, and Galileo scorers before accepting flags emitted by the app under test.
  • Choose Galileo metrics by failure contract, not by one global list.
  • Identify metric gaps before accepting a cost or quality change.

Gotchas

  • Agent self-reported flags are instrumentation breadcrumbs, not acceptance evidence.
  • Final citations alone are not enough when retrieved-source authority is part of the risk.
  • Do not infer latency, wall time, tokens, or cost as quality metrics.

Validation Loop

Before finalizing a metric profile, check it against skills/eval-engineer/references/metric-profile-checklist.md and confirm the profile names quality gates, cost/performance metrics, segment gates, metric direction, and known gaps.

Output

Findings first. Start with the highest-risk metric gaps and the keep/reject/ inconclusive measurement decision, then produce a metric profile or explain what evidence is missing. Do not improve the app until the expected-output contract and acceptance gates are clear.