agent-observability-experiment-analyzer
Analyze LLM experiment results. Handles single or comparative experiments, exploratory or Q&A modes. Use when user says "analyze experiment", "compare experiments", "analyze against baseline", or provides one or two experiment IDs for analysis.
datadog-labs · 已有 AI 审核记录
安装与使用
运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。
npx skills add datadog-labs/agent-skills --skill agent-observability-experiment-analyzer技能说明
name: agent-observability-experiment-analyzer description: Analyze LLM experiment results. Handles single or comparative experiments, exploratory or Q&A modes. Use when user says "analyze experiment", "compare experiments", "analyze against baseline", or provides one or two experiment IDs for analysis.
Backend
Detection — At the start of every invocation, before taking any action, determine which backend to use:
- If the user passed
--backend pupanywhere in their invocation → use pup mode immediately, regardless of whether MCP tools are present. Skip steps 2–4. - Check whether MCP tools are present in your active tool list. The canonical signal is whether a tool named
get_llmobs_experiment_summary(with or without themcp__datadog-llmo-mcp__prefix) appears in your available tools. - If MCP tools are present → use MCP mode throughout. Tool name binding: note the exact name under which
get_llmobs_experiment_summaryappears in your active tool list and use that exact name (prefixed or unprefixed) for every MCP tool call this invocation. All other experiment tools follow the same naming convention. - If MCP tools are absent → check whether
pupis executable: runpup --versionvia Bash. A JSON response containing"version"confirms pup is available. - If pup responds → use pup mode throughout. Translate every MCP tool call to its pup equivalent using the Tool Reference appendix at the bottom of this file.
- If neither is available → stop and tell the user:
"Neither the Datadog MCP server nor the pup CLI is available. Connect the MCP server (
claude mcp add --scope user --transport http datadog-llmo-mcp 'https://mcp.datadoghq.com/api/unstable/mcp-server/mcp?toolsets=llmobs') or install pup."
--backend pup is accepted anywhere in the invocation arguments and is stripped before passing remaining args to the skill logic.
pup invocation rules:
- Invoke via Bash:
pup llm-obs <subcommand> [flags] - pup always outputs JSON. Parse directly — no content-block unwrapping (unlike MCP results, which may wrap JSON in
[{"type": "text", "text": "<json>"}]). - If pup returns an auth error, tell the user to run
pup auth loginand stop. - Parallelization: issue multiple Bash tool calls in a single message (one pup command per call).
Invocation ID: At the very start of each invocation, before any MCP tool call, generate an 8-character hex invocation ID (e.g., 3a9f1c2b). Keep it constant for the entire invocation.
Intent tagging: On every MCP tool call, prefix telemetry.intent with skill:agent-observability-experiment-analyzer[<inv_id>] — followed by a description of why the tool is being called. On the first MCP tool call only, use skill:agent-observability-experiment-analyzer:start[<inv_id>] — instead (note the :start suffix). Example first call: skill:agent-observability-experiment-analyzer:start[3a9f1c2b] — Phase 1: get experiment summary to orient analysis
Unified Experiment Analyzer
Analyzes one or two LLM experiments. Supports four modes based on inputs:
| Inputs | Mode |
|---|---|
| 2 IDs, no question | Comparative Exploratory |
| 2 IDs + question | Comparative Q&A |
| 1 ID, no question | Single Exploratory |
| 1 ID + question | Single Q&A |
Usage
/agent-observability-experiment-analyzer <experiment_id_1> [experiment_id_2] [question text] [--output agent|file|notebook]
Arguments: $ARGUMENTS
Available Tools
Note: Tool names below are shown with an example
mcp__<server>__prefix. The actual prefix depends on the environment and MCP server name — it may differ or be absent entirely. Always call tools using the exact name visible in your active tool list (see Backend Detection step 3).
| Tool | Purpose |
|---|---|
mcp__datadog-llmo-mcp__get_llmobs_experiment_summary | Get total events, error count, metrics stats, available dimensions |
mcp__datadog-llmo-mcp__list_llmobs_experiment_events | Query events with filters, sorting, pagination |
mcp__datadog-llmo-mcp__get_llmobs_experiment_event | Get full event details (input, output, expected_output, metrics) |
mcp__datadog-llmo-mcp__get_llmobs_experiment_metric_values | Get metric stats overall and segmented by dimension. Use segment_by_dimension (not segment_dimension) to segment; optionally segment_dimension_value to filter to a specific value. |
mcp__datadog-llmo-mcp__get_llmobs_experiment_dimension_values | List unique values for a dimension with counts |
mcp__datadog-mcp-core__create_datadog_notebook | Export report as a Datadog notebook |
Phase 0 — Mode & Output Resolution
Parse $ARGUMENTS:
- Extract one or two UUID-format strings as experiment IDs (first = baseline/primary, second = candidate).
- Extract
--output agent|file|notebookflag if present. - The remaining text (after IDs and flags) is the question, if any.
Mode determination:
- 2 IDs + question → Comparative Q&A
- 2 IDs, no question → Comparative Exploratory
- 1 ID + question → Single Q&A
- 1 ID, no question → Single Exploratory
Output mode determination:
If --output was provided in arguments, use that mode and skip asking.
Otherwise, ask two separate sequential AskUserQuestion calls before proceeding — never combined into a single call:
- Analysis type: If no question text was provided in the arguments, ask whether the user wants exploratory analysis or has a specific question. Skip this call only if the user's intent is already clear from context (e.g. they typed a question alongside the IDs).
- Output destination: If
--outputwas not specified, ask where to deliver the report (chat, file, or Datadog notebook). Always ask this as its own standalone call.
Output modes:
- Agent (default): Display the full report in the conversation.
- File: Before starting, propose a path:
evals/reports/YYYY-MM-DD-<experiment-slug>-analysis.mdPresent it to the us