开发与测试
ai-coding-agents-observability-evals
Designs coding-agent observability and evals. Use when measuring traces, replay, checkpoint lineage, quality trajectories, tool grading, regression, or cost.
vasilyu1983 · 已有 AI 审核记录
安装与使用
运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。
npx skills add vasilyu1983/AI-Agents-public --skill ai-coding-agents-observability-evals技能说明
name: ai-coding-agents-observability-evals description: "Designs coding-agent observability and evals. Use when measuring traces, replay, checkpoint lineage, quality trajectories, tool grading, regression, or cost." compatibility: Portable core. Works on Claude Code and Codex. version: "1.2" last_validated: 2026-08-21
AI Coding Agents Observability And Evals
Use this skill to design or review the feedback loop around a coding-agent runtime: traces, replayable transcripts, eval packs, regression gates, tool-call grading, latency and cost accounting, and production failure triage.
This skill covers how you operate a coding-agent product after the core runtime exists. It does not replace the runtime skills themselves.
ASCII Flow
agent session
|
v
trace events
prompts + model turns + tool calls + permissions + file diffs + costs
|
v
replayable transcript
stable IDs + redaction + source/runtime correlation
|
v
eval pack
golden tasks + graders + regression gates + cost/latency budgets
|
v
release decision
pass | investigate | rollback | update eval coverage
Quick Reference
| Question | Read | Outcome |
|---|---|---|
| What should the trace and telemetry model include? | references/trace-and-telemetry-model.md | Durable trace schema, session correlation, event stages, and replay boundaries |
| How should evals, regressions, and cost controls work? | references/evals-regression-and-cost-ops.md | Golden tasks, iterative self-extension packs, trajectory scorecards, and cost-aware release gates |
| How do I use the eval/trace substrate to improve the harness itself? | references/harness-self-evolution.md | Closed-loop harness evolution: three observability pillars, falsifiable-contract edits, attribution |
| How does OpenAI Codex combine rollout replay, SQLite state, doctor reports, and telemetry? | references/openai-codex-rollout-doctor-telemetry.md | Replay artifacts, rebuildable state indexes, redacted diagnostics, W3C traces, token metrics |
| How does Codex wire OTel exporters and what analytics events exist? | references/openai-codex-otel-config.md | OtelSettings TOML schema, exporter selection, W3C tracestate, contrast with proprietary analytics events |
When To Use
- Design tracing and replay for a coding-agent CLI
- Add regression evals for coding, review, or task-execution agents
- Evaluate whether a coding agent preserves correctness and structural quality while extending its own workspace across evolving specifications
- Grade tool calls, patch quality, verification behavior, or handoff quality
- Build latency, token, and cost accounting for agent sessions
- Review how incidents and bad runs should be debugged from stored traces
Use Other Skills
| Need | Use Instead |
|---|---|
| Broader coding-agent architecture | ../ai-coding-agents/SKILL.md |
| Session persistence and transcript restore | ../ai-coding-agents-sessions/SKILL.md |
| Tool runtime design | ../ai-coding-agents-tools/SKILL.md |
| Generic agent eval harnesses | ../qa-agent-testing/SKILL.md |
| Reliability and observability outside agent systems | ../qa-observability/SKILL.md |
Default Workflow
- Define the trace spine. Session, turn, tool call, approval, worker, and verification events should share one correlation model.
- Store replay-safe artifacts. Persist prompts, tool inputs, outputs, diffs, approvals, and synthesized summaries with enough structure to replay failures.
- Separate product telemetry from eval telemetry. Production traces describe what happened; eval runs describe whether it was acceptable.
- Build golden task packs. Keep a representative set of coding, review, debugging, multi-agent, and iterative self-extension tasks with stable scoring rubrics.
- Grade behavior, not just final output. Score tool choice, verification discipline, retry loops, escalation quality, and cost efficiency.
- Keep telemetry cardinality under control. Stable prompt IDs, opaque hashes, and bounded error categories belong in event payloads; high-cardinality strings do not belong in metrics dimensions.
- Attach release gates to deltas. Compare candidate changes against a known baseline for quality, cost, latency, failure-mode drift, and—when work carries across checkpoints—trajectory slope and late-checkpoint regressions.
- Instrument incident triage. A bad run should be trace-searchable by repo, user, session, tool, provider, worker, and error family.
- Review regressions continuously. Add new real failures back into the eval corpus so the system hardens over time.
Host Rules
- Keep one canonical trace ID across the entire session lifecycle.
- Preserve causal order for tool calls, approvals, worker messages, and verification passes.
- Keep event ordering monotonic within a session even when log sinks or transports are asynchronous.
- Store enough normalized state to debug a run without depending on transient UI rendering.
- Score traces at multiple layers: final answer, tool behavior, and workflow correctness.
- Preserve checkpoint and workspace lineage for iterative evals; a final snapshot cannot explain when extensibility was lost.
- Track token and cost usage per turn and per subsystem, not only per session total.
- Use eval results to block releases when quality or cost drift exceeds explicit thresholds.
- Hash or redact user-identifying plugin or extension data before it becomes telemetry dimensions.
Scratch-Rebuild Coverage
- Coverage strength: strong for