跳至正文
OpenAgentSkill
浏览已开放 · 目录记录日期 2026-11-21 · 自动更新和账户服务暂时关闭。
AI 与知识库

agent-observability-experiment-analyzer

Analyze LLM experiment results. Handles single or comparative experiments, exploratory or Q&A modes. Use when user says "analyze experiment", "compare experiments", "analyze against baseline", or provides one or two experiment IDs for analysis.

datadog-labs · 已有 AI 审核记录

安装与使用

运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。

npx skills add datadog-labs/agent-skills --skill agent-observability-experiment-analyzer

技能说明


name: agent-observability-experiment-analyzer description: Analyze LLM experiment results. Handles single or comparative experiments, exploratory or Q&A modes. Use when user says "analyze experiment", "compare experiments", "analyze against baseline", or provides one or two experiment IDs for analysis.

Backend

Detection — At the start of every invocation, before taking any action, determine which backend to use:

  1. If the user passed --backend pup anywhere in their invocation → use pup mode immediately, regardless of whether MCP tools are present. Skip steps 2–4.
  2. Check whether MCP tools are present in your active tool list. The canonical signal is whether a tool named get_llmobs_experiment_summary (with or without the mcp__datadog-llmo-mcp__ prefix) appears in your available tools.
  3. If MCP tools are present → use MCP mode throughout. Tool name binding: note the exact name under which get_llmobs_experiment_summary appears in your active tool list and use that exact name (prefixed or unprefixed) for every MCP tool call this invocation. All other experiment tools follow the same naming convention.
  4. If MCP tools are absent → check whether pup is executable: run pup --version via Bash. A JSON response containing "version" confirms pup is available.
  5. If pup responds → use pup mode throughout. Translate every MCP tool call to its pup equivalent using the Tool Reference appendix at the bottom of this file.
  6. If neither is available → stop and tell the user:

    "Neither the Datadog MCP server nor the pup CLI is available. Connect the MCP server (claude mcp add --scope user --transport http datadog-llmo-mcp 'https://mcp.datadoghq.com/api/unstable/mcp-server/mcp?toolsets=llmobs') or install pup."

--backend pup is accepted anywhere in the invocation arguments and is stripped before passing remaining args to the skill logic.

pup invocation rules:

  • Invoke via Bash: pup llm-obs <subcommand> [flags]
  • pup always outputs JSON. Parse directly — no content-block unwrapping (unlike MCP results, which may wrap JSON in [{"type": "text", "text": "<json>"}]).
  • If pup returns an auth error, tell the user to run pup auth login and stop.
  • Parallelization: issue multiple Bash tool calls in a single message (one pup command per call).

Invocation ID: At the very start of each invocation, before any MCP tool call, generate an 8-character hex invocation ID (e.g., 3a9f1c2b). Keep it constant for the entire invocation.

Intent tagging: On every MCP tool call, prefix telemetry.intent with skill:agent-observability-experiment-analyzer[<inv_id>] — followed by a description of why the tool is being called. On the first MCP tool call only, use skill:agent-observability-experiment-analyzer:start[<inv_id>] — instead (note the :start suffix). Example first call: skill:agent-observability-experiment-analyzer:start[3a9f1c2b] — Phase 1: get experiment summary to orient analysis

Unified Experiment Analyzer

Analyzes one or two LLM experiments. Supports four modes based on inputs:

InputsMode
2 IDs, no questionComparative Exploratory
2 IDs + questionComparative Q&A
1 ID, no questionSingle Exploratory
1 ID + questionSingle Q&A

Usage

/agent-observability-experiment-analyzer <experiment_id_1> [experiment_id_2] [question text] [--output agent|file|notebook]

Arguments: $ARGUMENTS

Available Tools

Note: Tool names below are shown with an example mcp__<server>__ prefix. The actual prefix depends on the environment and MCP server name — it may differ or be absent entirely. Always call tools using the exact name visible in your active tool list (see Backend Detection step 3).

ToolPurpose
mcp__datadog-llmo-mcp__get_llmobs_experiment_summaryGet total events, error count, metrics stats, available dimensions
mcp__datadog-llmo-mcp__list_llmobs_experiment_eventsQuery events with filters, sorting, pagination
mcp__datadog-llmo-mcp__get_llmobs_experiment_eventGet full event details (input, output, expected_output, metrics)
mcp__datadog-llmo-mcp__get_llmobs_experiment_metric_valuesGet metric stats overall and segmented by dimension. Use segment_by_dimension (not segment_dimension) to segment; optionally segment_dimension_value to filter to a specific value.
mcp__datadog-llmo-mcp__get_llmobs_experiment_dimension_valuesList unique values for a dimension with counts
mcp__datadog-mcp-core__create_datadog_notebookExport report as a Datadog notebook

Phase 0 — Mode & Output Resolution

Parse $ARGUMENTS:

  1. Extract one or two UUID-format strings as experiment IDs (first = baseline/primary, second = candidate).
  2. Extract --output agent|file|notebook flag if present.
  3. The remaining text (after IDs and flags) is the question, if any.

Mode determination:

  • 2 IDs + question → Comparative Q&A
  • 2 IDs, no question → Comparative Exploratory
  • 1 ID + question → Single Q&A
  • 1 ID, no question → Single Exploratory

Output mode determination:

If --output was provided in arguments, use that mode and skip asking.

Otherwise, ask two separate sequential AskUserQuestion calls before proceeding — never combined into a single call:

  1. Analysis type: If no question text was provided in the arguments, ask whether the user wants exploratory analysis or has a specific question. Skip this call only if the user's intent is already clear from context (e.g. they typed a question alongside the IDs).
  2. Output destination: If --output was not specified, ask where to deliver the report (chat, file, or Datadog notebook). Always ask this as its own standalone call.

Output modes:

  1. Agent (default): Display the full report in the conversation.
  2. File: Before starting, propose a path: evals/reports/YYYY-MM-DD-<experiment-slug>-analysis.md Present it to the us
阅读完整来源 ↗