Skip to content
OpenAgentSkill
Browsing is open · Catalog captured 2026-11-21 · Automatic updates and account services are paused.
Development

ai-coding-agents-observability-evals

Designs coding-agent observability and evals. Use when measuring traces, replay, checkpoint lineage, quality trajectories, tool grading, regression, or cost.

vasilyu1983 · AI review on file

Installation

Read the source and dependencies before running a command. Copying does not install or execute anything.

npx skills add vasilyu1983/AI-Agents-public --skill ai-coding-agents-observability-evals

About this skill


name: ai-coding-agents-observability-evals description: "Designs coding-agent observability and evals. Use when measuring traces, replay, checkpoint lineage, quality trajectories, tool grading, regression, or cost." compatibility: Portable core. Works on Claude Code and Codex. version: "1.2" last_validated: 2026-08-21

AI Coding Agents Observability And Evals

Use this skill to design or review the feedback loop around a coding-agent runtime: traces, replayable transcripts, eval packs, regression gates, tool-call grading, latency and cost accounting, and production failure triage.

This skill covers how you operate a coding-agent product after the core runtime exists. It does not replace the runtime skills themselves.

ASCII Flow

agent session
  |
  v
trace events
  prompts + model turns + tool calls + permissions + file diffs + costs
  |
  v
replayable transcript
  stable IDs + redaction + source/runtime correlation
  |
  v
eval pack
  golden tasks + graders + regression gates + cost/latency budgets
  |
  v
release decision
  pass | investigate | rollback | update eval coverage

Quick Reference

QuestionReadOutcome
What should the trace and telemetry model include?references/trace-and-telemetry-model.mdDurable trace schema, session correlation, event stages, and replay boundaries
How should evals, regressions, and cost controls work?references/evals-regression-and-cost-ops.mdGolden tasks, iterative self-extension packs, trajectory scorecards, and cost-aware release gates
How do I use the eval/trace substrate to improve the harness itself?references/harness-self-evolution.mdClosed-loop harness evolution: three observability pillars, falsifiable-contract edits, attribution
How does OpenAI Codex combine rollout replay, SQLite state, doctor reports, and telemetry?references/openai-codex-rollout-doctor-telemetry.mdReplay artifacts, rebuildable state indexes, redacted diagnostics, W3C traces, token metrics
How does Codex wire OTel exporters and what analytics events exist?references/openai-codex-otel-config.mdOtelSettings TOML schema, exporter selection, W3C tracestate, contrast with proprietary analytics events

When To Use

  • Design tracing and replay for a coding-agent CLI
  • Add regression evals for coding, review, or task-execution agents
  • Evaluate whether a coding agent preserves correctness and structural quality while extending its own workspace across evolving specifications
  • Grade tool calls, patch quality, verification behavior, or handoff quality
  • Build latency, token, and cost accounting for agent sessions
  • Review how incidents and bad runs should be debugged from stored traces

Use Other Skills

NeedUse Instead
Broader coding-agent architecture../ai-coding-agents/SKILL.md
Session persistence and transcript restore../ai-coding-agents-sessions/SKILL.md
Tool runtime design../ai-coding-agents-tools/SKILL.md
Generic agent eval harnesses../qa-agent-testing/SKILL.md
Reliability and observability outside agent systems../qa-observability/SKILL.md

Default Workflow

  1. Define the trace spine. Session, turn, tool call, approval, worker, and verification events should share one correlation model.
  2. Store replay-safe artifacts. Persist prompts, tool inputs, outputs, diffs, approvals, and synthesized summaries with enough structure to replay failures.
  3. Separate product telemetry from eval telemetry. Production traces describe what happened; eval runs describe whether it was acceptable.
  4. Build golden task packs. Keep a representative set of coding, review, debugging, multi-agent, and iterative self-extension tasks with stable scoring rubrics.
  5. Grade behavior, not just final output. Score tool choice, verification discipline, retry loops, escalation quality, and cost efficiency.
  6. Keep telemetry cardinality under control. Stable prompt IDs, opaque hashes, and bounded error categories belong in event payloads; high-cardinality strings do not belong in metrics dimensions.
  7. Attach release gates to deltas. Compare candidate changes against a known baseline for quality, cost, latency, failure-mode drift, and—when work carries across checkpoints—trajectory slope and late-checkpoint regressions.
  8. Instrument incident triage. A bad run should be trace-searchable by repo, user, session, tool, provider, worker, and error family.
  9. Review regressions continuously. Add new real failures back into the eval corpus so the system hardens over time.

Host Rules

  • Keep one canonical trace ID across the entire session lifecycle.
  • Preserve causal order for tool calls, approvals, worker messages, and verification passes.
  • Keep event ordering monotonic within a session even when log sinks or transports are asynchronous.
  • Store enough normalized state to debug a run without depending on transient UI rendering.
  • Score traces at multiple layers: final answer, tool behavior, and workflow correctness.
  • Preserve checkpoint and workspace lineage for iterative evals; a final snapshot cannot explain when extensibility was lost.
  • Track token and cost usage per turn and per subsystem, not only per session total.
  • Use eval results to block releases when quality or cost drift exceeds explicit thresholds.
  • Hash or redact user-identifying plugin or extension data before it becomes telemetry dimensions.

Scratch-Rebuild Coverage

  • Coverage strength: strong for
Read the complete source ↗