agent-observability-replay-trace
Use when a developer wants to iterate on ONE specific Agent Observability / LLM Obs trace whose output they didn't like — re-running that trace against their LOCAL code, seeing a concise diff of the old vs new output, and looping (change code → replay → diff) until satisfied. Invoked as /agent-observability-replay-trace <trace-id> [changes to test]. Signals: "replay this trace"; "iterate on a trace"; "this trace's output is wrong, fix it and re-run"; "re-run trace <id> with <change>"; pasting a trace id from the Agent Observability UI with a description of what to fix. It fetches the trace via the datadog-llmo MCP or the pup CLI, edits code, re-runs the app to emit a NEW trace, and diffs the two — no local server, no browser. For agents traced with ddtrace / LLM Obs (Python first-class), with JSON-serializable entry input. Do NOT use for: scored Experiments or the browser "Replay" button (that's agent-observability-replay-experiment), building an experiment from a dataset/CSV, writing
datadog-labs · 已有 AI 审核记录
安装与使用
运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。
npx skills add datadog-labs/agent-skills --skill agent-observability-replay-trace技能说明
name: agent-observability-replay-trace description: >- Use when a developer wants to iterate on ONE specific Agent Observability / LLM Obs trace whose output they didn't like — re-running that trace against their LOCAL code, seeing a concise diff of the old vs new output, and looping (change code → replay → diff) until satisfied. Invoked as /agent-observability-replay-trace [changes to test]. Signals: "replay this trace"; "iterate on a trace"; "this trace's output is wrong, fix it and re-run"; "re-run trace with "; pasting a trace id from the Agent Observability UI with a description of what to fix. It fetches the trace via the datadog-llmo MCP or the pup CLI, edits code, re-runs the app to emit a NEW trace, and diffs the two — no local server, no browser. For agents traced with ddtrace / LLM Obs (Python first-class), with JSON-serializable entry input. Do NOT use for: scored Experiments or the browser "Replay" button (that's agent-observability-replay-experiment), building an experiment from a dataset/CSV, writing evaluators, root-causing failed traces, or RUM/HTTP session replay.
Replay a trace against local code
A fast iteration loop on a single production trace: take a trace whose output a developer didn't like, optionally change the code, re-run it against their LOCAL code, and show a concise diff of old vs new output — repeating until they're happy. Assumes nothing about the project's layout.
Invoked from the developer's coding agent: /agent-observability-replay-trace <trace-id> [<changes to test>].
With no modification, do the replay + diff only (a reproduce/regression check), then offer to enter the loop.
This file is the workflow spine — terse on purpose. The depth lives in references/details.md (trace
backend + pup flags, the runner contract, export mode, polling, the trace-link scoping fix) and
references/local-setup.md (making a deployed-only app locally runnable). Read details.md before you touch
pup or generate the runner.
Writing code — keep comments minimal to none. Everything you generate or edit (the annotation, the
runner's ENTRYPOINTS entries, a local harness, iteration edits) should match the surrounding code and
carry no unnecessary comments — don't narrate what the code plainly does; add a comment only for a
genuinely non-obvious why.
Intent tagging: On every datadog-llmo MCP tool call, prefix telemetry.intent with skill:agent-observability-replay-trace[<inv_id>] — (a short per-run id, generated once and reused for every call) followed by a description of why the tool is being called. On the first MCP tool call only, use skill:agent-observability-replay-trace:start[<inv_id>] — instead (note the :start suffix). Example first call: skill:agent-observability-replay-trace:start[3a9f1c2b] — fetch the original trace's baseline output. pup-CLI calls carry no telemetry.intent, so this applies only on the MCP path.
Interaction model — selector gates, never a hard stop
This is a live loop. At every decision point present the choices as an AskUserQuestion selector (the
plan-mode-style menu), not a plain question that ends your turn. Two gates: (a) after you propose code
changes, before replaying; (b) after each diff. The selector's free-text option lets the user type detail
(what to refine) inline — act on it directly, don't ask a follow-up. Keep re-presenting after every replay
until they pick "stop here".
Scope — check first
- Traced with
ddtrace/ LLM Obs (anml_app+ a discoverable entrypoint). Python is first-class; other languages work but you write the runner to the contract in their SDK/build tooling. - JSON-serializable entrypoint input, and a callable seam for the root span (see step 3.5 — not a binary "is it runnable?"; deployed-only apps often still expose a plain callable).
- A trace-access backend — the
datadog-llmoMCP (used when present) or thepupCLI (fallback, and the easier install if you have neither) (step 0). - Credentials:
DD_API_KEY+DD_SITE+ provider key(s). NotDD_APP_KEY— plain trace, not an Experiment (that'sagent-observability-replay-experiment). - Side effects, irreversible: replaying re-runs real code (model spend + real writes), and LLM Obs
traces cannot be deleted — a mis-scoped replay (wrong ml_app) permanently pollutes the production app's
dashboards/eval sets. That's why the
<ml_app>-localisolation (steps 4/6/7) is load-bearing, not tidy. Warn before the first replay.
Workflow
0. Ensure a trace-access backend
Pick, in order: (1) the MCP if mcp__datadog-llmo-mcp__* tools are present — the default (slightly
richer for reads: structured tree + content_info); (2) else pup if installed and pup auth targets
the app's org; (3) else the user has neither → guide the pup install (it's easier to set up than the
MCP, so recommend pup here):
brew tap datadog-labs/pack && brew install datadog-labs/pack/pup
pup auth login
(MCP alternative: claude mcp add --scope user --transport http "datadog-llmo-mcp" "https://mcp.datadoghq.com/api/unstable/mcp-server/mcp?toolsets=llmobs"; see https://docs.datadoghq.com/bits_ai/mcp_server/setup/.) Don't proceed without a backend.
The backend↔operation mapping and pup's exact flags/gotchas are in details.md — read that section before
using pup. Two pup musts: (1) results come back at data.spans[] or top-level spans[]
(varies by version/--no-agent) — parse whichever is present, or you get zero hits on an ingested
trace (a silent false negative, step 7); (2) check token expiry (pup auth status), not just that auth
exists — expiry mid-loop looks like "trace not found."
1. Parse the command
<trace-id> + optional free-text modification (everything after the id); none → diff-only mode. Determine
the ml_app from the project (`LLMObs.enable(ml_a