跳至正文
OpenAgentSkill
浏览已开放 · 目录记录日期 2026-11-21 · 自动更新和账户服务暂时关闭。
浏览器与自动化

agent-observability-replay-trace

Use when a developer wants to iterate on ONE specific Agent Observability / LLM Obs trace whose output they didn't like — re-running that trace against their LOCAL code, seeing a concise diff of the old vs new output, and looping (change code → replay → diff) until satisfied. Invoked as /agent-observability-replay-trace <trace-id> [changes to test]. Signals: "replay this trace"; "iterate on a trace"; "this trace's output is wrong, fix it and re-run"; "re-run trace <id> with <change>"; pasting a trace id from the Agent Observability UI with a description of what to fix. It fetches the trace via the datadog-llmo MCP or the pup CLI, edits code, re-runs the app to emit a NEW trace, and diffs the two — no local server, no browser. For agents traced with ddtrace / LLM Obs (Python first-class), with JSON-serializable entry input. Do NOT use for: scored Experiments or the browser "Replay" button (that's agent-observability-replay-experiment), building an experiment from a dataset/CSV, writing

datadog-labs · 已有 AI 审核记录

安装与使用

运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。

npx skills add datadog-labs/agent-skills --skill agent-observability-replay-trace

技能说明


name: agent-observability-replay-trace description: >- Use when a developer wants to iterate on ONE specific Agent Observability / LLM Obs trace whose output they didn't like — re-running that trace against their LOCAL code, seeing a concise diff of the old vs new output, and looping (change code → replay → diff) until satisfied. Invoked as /agent-observability-replay-trace [changes to test]. Signals: "replay this trace"; "iterate on a trace"; "this trace's output is wrong, fix it and re-run"; "re-run trace with "; pasting a trace id from the Agent Observability UI with a description of what to fix. It fetches the trace via the datadog-llmo MCP or the pup CLI, edits code, re-runs the app to emit a NEW trace, and diffs the two — no local server, no browser. For agents traced with ddtrace / LLM Obs (Python first-class), with JSON-serializable entry input. Do NOT use for: scored Experiments or the browser "Replay" button (that's agent-observability-replay-experiment), building an experiment from a dataset/CSV, writing evaluators, root-causing failed traces, or RUM/HTTP session replay.

Replay a trace against local code

A fast iteration loop on a single production trace: take a trace whose output a developer didn't like, optionally change the code, re-run it against their LOCAL code, and show a concise diff of old vs new output — repeating until they're happy. Assumes nothing about the project's layout.

Invoked from the developer's coding agent: /agent-observability-replay-trace <trace-id> [<changes to test>]. With no modification, do the replay + diff only (a reproduce/regression check), then offer to enter the loop.

This file is the workflow spine — terse on purpose. The depth lives in references/details.md (trace backend + pup flags, the runner contract, export mode, polling, the trace-link scoping fix) and references/local-setup.md (making a deployed-only app locally runnable). Read details.md before you touch pup or generate the runner.

Writing code — keep comments minimal to none. Everything you generate or edit (the annotation, the runner's ENTRYPOINTS entries, a local harness, iteration edits) should match the surrounding code and carry no unnecessary comments — don't narrate what the code plainly does; add a comment only for a genuinely non-obvious why.

Intent tagging: On every datadog-llmo MCP tool call, prefix telemetry.intent with skill:agent-observability-replay-trace[<inv_id>] — (a short per-run id, generated once and reused for every call) followed by a description of why the tool is being called. On the first MCP tool call only, use skill:agent-observability-replay-trace:start[<inv_id>] — instead (note the :start suffix). Example first call: skill:agent-observability-replay-trace:start[3a9f1c2b] — fetch the original trace's baseline output. pup-CLI calls carry no telemetry.intent, so this applies only on the MCP path.

Interaction model — selector gates, never a hard stop

This is a live loop. At every decision point present the choices as an AskUserQuestion selector (the plan-mode-style menu), not a plain question that ends your turn. Two gates: (a) after you propose code changes, before replaying; (b) after each diff. The selector's free-text option lets the user type detail (what to refine) inline — act on it directly, don't ask a follow-up. Keep re-presenting after every replay until they pick "stop here".

Scope — check first

  • Traced with ddtrace / LLM Obs (an ml_app + a discoverable entrypoint). Python is first-class; other languages work but you write the runner to the contract in their SDK/build tooling.
  • JSON-serializable entrypoint input, and a callable seam for the root span (see step 3.5 — not a binary "is it runnable?"; deployed-only apps often still expose a plain callable).
  • A trace-access backend — the datadog-llmo MCP (used when present) or the pup CLI (fallback, and the easier install if you have neither) (step 0).
  • Credentials: DD_API_KEY + DD_SITE + provider key(s). Not DD_APP_KEY — plain trace, not an Experiment (that's agent-observability-replay-experiment).
  • Side effects, irreversible: replaying re-runs real code (model spend + real writes), and LLM Obs traces cannot be deleted — a mis-scoped replay (wrong ml_app) permanently pollutes the production app's dashboards/eval sets. That's why the <ml_app>-local isolation (steps 4/6/7) is load-bearing, not tidy. Warn before the first replay.

Workflow

0. Ensure a trace-access backend

Pick, in order: (1) the MCP if mcp__datadog-llmo-mcp__* tools are present — the default (slightly richer for reads: structured tree + content_info); (2) else pup if installed and pup auth targets the app's org; (3) else the user has neither → guide the pup install (it's easier to set up than the MCP, so recommend pup here):

brew tap datadog-labs/pack && brew install datadog-labs/pack/pup
pup auth login

(MCP alternative: claude mcp add --scope user --transport http "datadog-llmo-mcp" "https://mcp.datadoghq.com/api/unstable/mcp-server/mcp?toolsets=llmobs"; see https://docs.datadoghq.com/bits_ai/mcp_server/setup/.) Don't proceed without a backend. The backend↔operation mapping and pup's exact flags/gotchas are in details.md — read that section before using pup. Two pup musts: (1) results come back at data.spans[] or top-level spans[] (varies by version/--no-agent) — parse whichever is present, or you get zero hits on an ingested trace (a silent false negative, step 7); (2) check token expiry (pup auth status), not just that auth exists — expiry mid-loop looks like "trace not found."

1. Parse the command

<trace-id> + optional free-text modification (everything after the id); none → diff-only mode. Determine the ml_app from the project (`LLMObs.enable(ml_a

阅读完整来源 ↗