agent-observability-session-classify
Classify whether user intent was satisfied in a Datadog Agent Observability trace or session. Three modes: (1) session_id — classify a single CMD+I assistant session with RUM; (2) trace_id — classify a single Agent Observability trace without RUM; (3) ml_app — sample and classify multiple sessions or traces from a given LLM app. Output is compact by default (verdict + one-sentence reason). Use when evaluating satisfaction, classifying sessions/traces, labeling data, or generating signal for agent-observability-eval-pipeline or agent-observability-trace-rca.
datadog-labs · 已有 AI 审核记录
安装与使用
运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。
npx skills add datadog-labs/agent-skills --skill agent-observability-session-classify技能说明
name: agent-observability-session-classify description: > Classify whether user intent was satisfied in a Datadog Agent Observability trace or session. Three modes: (1) session_id — classify a single CMD+I assistant session with RUM; (2) trace_id — classify a single Agent Observability trace without RUM; (3) ml_app — sample and classify multiple sessions or traces from a given LLM app. Output is compact by default (verdict + one-sentence reason). Use when evaluating satisfaction, classifying sessions/traces, labeling data, or generating signal for agent-observability-eval-pipeline or agent-observability-trace-rca.
Backend
Detection — At the start of every invocation, before taking any action, determine which backend to use:
- If the user passed
--backend pupanywhere in their invocation → use pup mode immediately, regardless of whether MCP tools are present. Skip steps 2–4. - Check whether MCP tools are present in your active tool list. The canonical signal is whether
mcp__datadog-llmo-mcp__search_llmobs_spansappears in your available tools. - If MCP tools are present → use MCP mode throughout. Call MCP tools exactly as named in this skill's workflow sections.
- If MCP tools are absent → check whether
pupis executable: runpup --versionvia Bash. A JSON response containing"version"confirms pup is available. - If pup responds → use pup mode throughout. Translate every MCP tool call to its pup equivalent using the Tool Reference appendix at the bottom of this file.
- If neither is available → stop and tell the user:
"Neither the Datadog MCP server nor the pup CLI is available. Connect the MCP server (
claude mcp add --scope user --transport http datadog-llmo-mcp 'https://mcp.datadoghq.com/api/unstable/mcp-server/mcp?toolsets=llmobs,rum') or install pup."
--backend pup is accepted anywhere in the invocation arguments and is stripped before passing remaining args to the skill logic.
pup invocation rules:
- Invoke via Bash:
pup llm-obs <subcommand> [flags] - pup always outputs JSON. Parse directly — no content-block unwrapping (unlike MCP results, which may wrap JSON in
[{"type": "text", "text": "<json>"}]). - If pup returns an auth error, tell the user to run
pup auth loginand stop. - Parallelization: issue multiple Bash tool calls in a single message (one pup command per call).
- Time flags: pup accepts bare duration strings (
1h,7d,30m) and RFC3339 timestamps. Do not usenow--prefixed strings — strip the prefix when converting from a skill--timeframeargument:now-7d→7d,now-24h→24h,now-30d→30d. --summaryonpup llm-obs spans searchstrips payload fields to essential metadata only. Use it in bulk/search phases where content is not needed.
pup mode notes by entry mode:
session_idmode: Steps 1–3 and Step 5 work fully. Step 4 (RUM) usespup rum aggregate --user-email EMAILinstead ofanalyze_rum_events— see Tool Reference. Step 4b (audit trail) is pup-native and queries the active user's own org via OAuth.trace_idmode: Full parity with MCP mode.ml_appmode: Option A (aggregate_spans) is unavailable in pup — skip it and proceed directly to Option B.
Invocation ID: At the very start of each invocation, before any MCP tool call, generate an 8-character hex invocation ID (e.g., 3a9f1c2b). Keep it constant for the entire invocation.
Intent tagging: On every MCP tool call, prefix telemetry.intent with skill:agent-observability-session-classify[<inv_id>] — followed by a description of why the tool is being called. On the first MCP tool call only, use skill:agent-observability-session-classify:start[<inv_id>] — instead (note the :start suffix). Example first call: skill:agent-observability-session-classify:start[3a9f1c2b] — Step 1: enumerate turn root spans for session abc-123
Skill: eval-session-classify
Classification skill for Datadog Agent Observability sessions and traces. Produces a satisfaction verdict (yes / partial / no) with a brief reasoning string. Designed to be called standalone or by eval-pipeline.
Inputs
| Input | Mode | Required | Description |
|---|---|---|---|
session_id | session_id mode | Yes | UUID of a Datadog CMD+I assistant session |
trace_id | trace_id mode | Yes | Trace ID from Agent Observability |
ml_app | ml_app mode | Yes | LLM app name to sample from |
timeframe | ml_app mode | No (default: now-7d) | How far back to sample |
sample_limit | ml_app mode | No (default: 20, cap: 50) | Number of sessions or traces to classify |
If none of session_id, trace_id, or ml_app is provided → stop immediately and return:
{
"error": "missing_input",
"detail": "Provide one of: session_id, trace_id, or ml_app."
}
Phase 0 — Mode Detection
session_idprovided → session_id mode → go to Session Modetrace_idprovided (nosession_id) → trace_id mode → go to Trace Modeml_appprovided (nosession_id, notrace_id) → ml_app mode → go to ML App Mode- Nothing provided → emit error (see above)
Output Format
Default (compact) — used in all modes unless the caller requests verbose:
verdict: yes | partial | no
reason: <one sentence>
Verbose — full markdown report (see end of each mode section). Request verbose by including verbose: true in input or asking for a detailed report.
In ml_app mode, a summary table is always appended after the per-trace compact blocks.
Content Retrieval Cascade
Reading conversation content follows this cascade across all modes. Run the cascade for every turn root span identified in Step 1 (session mode) or Step T1 (trace mode) before forming a verdict.
阅读完整来源 ↗Completeness gate: do not proceed to RUM (session mode) or classification (any mode) until the cascade has run