跳至正文
OpenAgentSkill
浏览已开放 · 目录记录日期 2026-11-21 · 自动更新和账户服务暂时关闭。
设计与 UI

agent-evaluation-operations

Evaluate, observe, and release AI-agent workflows. Use for agent regression suites, prompt or model comparisons, scenario simulation, tool-call verification, traces, quality thresholds, cost and latency budgets, red-team cases, or release gates for AI systems.

TheGoat395 · 已有 AI 审核记录

安装与使用

运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。

npx skills add TheGoat395/Codex-Skills --skill agent-evaluation-operations

技能说明


name: agent-evaluation-operations description: Evaluate, observe, and release AI-agent workflows. Use for agent regression suites, prompt or model comparisons, scenario simulation, tool-call verification, traces, quality thresholds, cost and latency budgets, red-team cases, or release gates for AI systems.

Agent Evaluation Operations

Start from real failure modes

Create a test inventory covering ordinary success, ambiguous input, missing data, conflicting evidence, unavailable tool, duplicate event, unsafe external action, escalation, and recovery. For each case record input, allowed sources/tools, expected structured result, prohibited outcome, scorer, and acceptance threshold.

Measure the whole workflow

Evaluate factual grounding, action correctness, policy and approval adherence, tool selection, completion state, latency, cost, and recoverability. Use deterministic assertions whenever possible; use model grading only with a written rubric and calibration examples.

Trace run ID, model, prompt/version, tools, input class, outputs, error, latency, cost, and approval path. Redact or avoid sensitive payload capture by default.

Gate releases

  • Keep a small deterministic smoke suite for every change.
  • Keep regression cases for every confirmed failure.
  • Test external calls in sandbox or dry-run mode before production.
  • Fail release when required scenario, approval, cost, or quality thresholds are not met.
  • Separate test evidence from production evidence.

Use Promptfoo project-locally when its capabilities fit and its telemetry, remote-generation, configuration, and credentials have been reviewed. Do not claim safety or production readiness from one successful demo. Route system design to $agent-orchestration-architecture and run the repository's applicable security checks before release.