跳至正文
OpenAgentSkill
浏览已开放 · 目录记录日期 2026-11-21 · 自动更新和账户服务暂时关闭。
设计与 UI

a-b-test-design

Design an A/B experiment — hypothesis, variants, primary metric, and sample size. Use when a change can be measured quantitatively at scale. For observing behaviour qualitatively, use `test-scenario`.

aicodedecode · 已有静态检查记录

安装与使用

运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。

npx skills add aicodedecode/awesome-muse-skills --skill a-b-test-design

技能说明


name: a-b-test-design description: Design an A/B experiment — hypothesis, variants, primary metric, and sample size. Use when a change can be measured quantitatively at scale. For observing behaviour qualitatively, use test-scenario.

A/B Test Design

You are an expert in designing rigorous A/B experiments that produce actionable results.

What You Do

You design A/B tests with clear hypotheses, controlled variants, appropriate metrics, and statistical rigor.

Test Structure

1. Hypothesis

Structured as: 'If we [change], then [outcome] will [improve/decrease] because [rationale].'

2. Variants

  • Control (A): current design
  • Treatment (B): proposed change
  • Keep changes isolated — test one variable at a time

3. Primary Metric

The single most important measure of success. Must be measurable, relevant, and sensitive to the change.

4. Secondary Metrics

Supporting measures and guardrail metrics to detect unintended consequences.

5. Sample Size

Based on: minimum detectable effect, baseline conversion rate, statistical significance level (typically 95%), and power (typically 80%).

6. Duration

Run until sample size is reached. Account for weekly cycles (run in full weeks). Minimum 1-2 weeks typically.

Common Pitfalls

  • Peeking at results before completion
  • Too many variants at once
  • Metric not sensitive enough to detect change
  • Sample size too small
  • Not accounting for novelty effects
  • Ignoring segmentation effects

When Not to A/B Test

  • Very low traffic (insufficient sample)
  • Ethical concerns with withholding improvement
  • Foundational changes that affect everything
  • When qualitative insight is more valuable

Best Practices

  • One hypothesis per test
  • Document everything before starting
  • Don't stop early on positive results
  • Analyze segments after overall results
  • Share learnings broadly regardless of outcome