跳至正文
OpenAgentSkill
浏览已开放 · 目录记录日期 2026-11-21 · 自动更新和账户服务暂时关闭。
其他技能

review-experiment-results

Explain Harness FME experiment results: winner determination, statistical significance, guardrail metric impact, and data-quality caveats (low sample size, missing data). Classifies each metric's outcome and produces plain-language readout by default, surfacing underlying stats (p-value, CI, sample size) on request. Use when asked to explain, review, or interpret experiment results; "did treatment X win"; "is this experiment significant"; "what happened to metric Y in this experiment"; or "should we ship this experiment". Do NOT use for changing experiments (manage-experiments), choosing metrics (choose-metric), or diagnosing bias (SRM root cause, peeking, Simpson's paradox). Trigger phrases: experiment results, readout, winner, statistical significance, guardrail metric, ship this experiment.

Harness · 已有静态检查记录

安装与使用

运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。

npx skills add harness/harness-skills --skill review-experiment-results

技能说明


name: review-experiment-results description: >- Explain Harness FME experiment results: winner determination, statistical significance, guardrail metric impact, and data-quality caveats (low sample size, missing data). Classifies each metric's outcome and produces plain-language readout by default, surfacing underlying stats (p-value, CI, sample size) on request. Use when asked to explain, review, or interpret experiment results; "did treatment X win"; "is this experiment significant"; "what happened to metric Y in this experiment"; or "should we ship this experiment". Do NOT use for changing experiments (manage-experiments), choosing metrics (choose-metric), or diagnosing bias (SRM root cause, peeking, Simpson's paradox). Trigger phrases: experiment results, readout, winner, statistical significance, guardrail metric, ship this experiment. metadata: author: Harness version: 2.1.2 mcp-server: harness-mcp summary: Explain experiment results, including winner, significance, guardrail impact, and data-quality caveats license: Apache-2.0 compatibility: Requires the Harness MCP server or the Harness CLI

Review Experiment Results

Explain an FME experiment's outcome in plain language: whether there's a winner, why or why not, how guardrail metrics were affected, and any data-quality caveats - with underlying statistics available on request.

Related: manage-experiments (experiment changes), choose-metric (metric selection).

Tools

Works through the Harness MCP server or the Harness CLI; names are from tool-map.md.

OperationMCPCLI
List experimentsharness_list · fme_experiment · filters: { parent_type, name?, match_type: "contains", status?: ["ACTIVE", "PAUSED", "COMPLETED", "ARCHIVED"], offset: 0, limit: 100 } · compact: falseharness list experiment --parent-type FEATURE_FLAG [--search <name>] --status ACTIVE, then repeat with --status PAUSED, --status COMPLETED, --status ARCHIVED (CLI --status is single-value; API defaults to ACTIVE if omitted)
Get experimentharness_get · fme_experiment · params: { experiment_id }harness get experiment <experiment-id>
Get settingsharness_get · fme_experiment_settings · params: { experiment_id }harness get experiment:settings <experiment-id>
List resultsharness_list · fme_experiment_result · filters: { experiment_id, comparisons? }harness list experiment:results <experiment-id> --raw
Get/list metricsharness_get · fme_metric · params.metric_id (one call per id), or harness_list · fme_metric · filters: { ids } · compact: falseharness get metric <id> once per id, or harness list metric --id <id> once per id - the CLI's --id flag is repeatable but only the first value reaches the API (spec bug), so repeating --id id1 --id id2 on one call silently drops id2

Instructions

Phase 1: Establish scope

Follow scope-establishment.md.

Step 1: Resolve the experiment

If the user gives an exact experiment ID, Get experiment directly - skip list. Otherwise List experiments for parent type FEATURE_FLAG (or AI_CONFIG if user specified), with name substring match if provided (MCP: match_type: "contains"). Default parent type: FEATURE_FLAG.

If user didn't specify status, list ACTIVE, PAUSED, COMPLETED, and ARCHIVED - a finished or archived experiment is exactly the kind a results question is usually about, so never default to active-only here. Fully paginate all selected statuses per pagination before deciding uniqueness or absence; CLI needs a separate paginated query per status. More than one match or no name given → ask which experiment. If a scan is incomplete, report that instead of silently selecting a first-page match.

Step 2: Fetch experiment definition

Get experiment. Key fields: status, environment, startAt/endAt, baselineTreatment, comparisonTreatments[], keyMetrics[], supportingMetrics[].

keyMetrics/supportingMetrics are the only metric lists on the experiment. GUARDRAIL and ALERT metrics are workspace-wide categories applying to every experiment, so they never appear here - read role from each result's category (Step 4). Empty keyMetrics → no winner criterion; ask which metric(s) should drive verdict.

More than one comparisonTreatments and user didn't name one → report every treatment (one verdict each, see Output Format). If question presumes a single treatment ("did it win?") → ask which treatment or confirm per-treatment readout.

Step 3: Fetch experiment settings

Get settings. Always returns applied settings (404 only if experiment doesn't exist). source: "EXPERIMENT_OVERRIDE" means experiment has its own override; other values mean it inherits org defaults.

Fields to carry forward: significanceThreshold (not always 0.05), multipleComparisonCorrection (NONE or GROUPWISE_HOCHBERG; applies only to KEY/SUPPORTING, never GUARDRAIL/ALERT), minimumSampleSize (per treatment), statisticalTestType (FIXED_HORIZON/SEQUENTIAL), reviewPeriod (ISO-8601 duration), varianceReduction.method (NONE/CUPED).

Step 4: Fetch evaluated results

List results for the experiment, optionally narrowed to specific comparison treatments. List-only (no single-result get, no per-environment filter). Returns latest calculation run only, as one row per (metric, comparison treatment) pair, plus top-level calculatedAt (null if never calculated). If narrowing to specific treatments, filter the returned rows on each item's comparison (see tool-map.md).

Per-row fields: category (KEY, SUPPORTING, GUARDRAIL, or ALERT), `compariso

阅读完整来源 ↗