ab-testing
Use when designing or analyzing a controlled experiment — falsifiable hypothesis, sample size from an MDE, reading significance/CI/power, CUPED, or rescuing tests that won't go significant. NOT recurring metric tracking (that is `analytics`), NOT north-star/KPI trees (that is `kpi-framework`), NOT projecting metrics forward (that is `forecasting`).
ericrisco · 已有 AI 审核记录
安装与使用
运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。
npx skills add ericrisco/rsc-harness --skill ab-testing技能说明
name: ab-testing
description: "Use when designing or analyzing a controlled experiment — falsifiable hypothesis, sample size from an MDE, reading significance/CI/power, CUPED, or rescuing tests that won't go significant. NOT recurring metric tracking (that is analytics), NOT north-star/KPI trees (that is kpi-framework), NOT projecting metrics forward (that is forecasting)."
tags: [ab-testing, experimentation, statistics, cuped, sample-size, hypothesis-testing]
recommends: [analytics, kpi-framework, forecasting, data-cleaning, python, reporting]
origin: risco
A/B testing — design and read a defensible experiment
An experiment without a pre-committed sample size and a single primary metric is not an experiment. It is a dashboard you stare at until it tells you what you wanted to hear. The discipline lives almost entirely before traffic ships: a falsifiable hypothesis, one primary metric, a sample size derived from the smallest effect worth detecting, and a stop rule you cannot renegotiate at 2pm on day four.
Pre-test checklist — every line true before any traffic
Each one is a place experiments die silently.
- A falsifiable hypothesis — names the change, the direction, and the metric it moves.
- Exactly ONE primary metric. More than one primary = multiple comparisons = inflated false positives.
- Guardrail metrics — what you refuse to harm (latency, refunds, unsubscribes) even for a win.
- The randomization unit = the analysis unit (usually the user). Mixing them is pseudoreplication.
- An MDE — the smallest lift that would change a decision. Not "any difference."
- A computed sample size and the duration it implies at your real daily eligible traffic.
- A fixed stop rule — a date or an n you commit to before launch. No "we'll see how it looks."
Step 1 — Hypothesis and metrics
State a null you can reject. "The new checkout button changes purchase conversion" with H0: conversion equal across arms, H1: it differs. Vague aspirations ("improve the funnel") have no rejection region.
Pick one primary metric and freeze it. Why: every extra primary metric is another coin flip at α, so three "primary" metrics turn a 5% false-positive rate into roughly 14%. Demote the rest to secondary.
Randomize on the same unit you analyze on. If a user sees the variant on every visit, randomize by user, not by session — analyzing 50k sessions from 8k users treats correlated observations as independent and fabricates significance.
Bad: "We think the redesign will improve engagement and revenue and retention." (no null, 3 primaries, no number)
Good: "H0: 30-day purchase conversion is equal between control and the new one-click button.
H1: it differs. Primary: purchase conversion. Guardrails: refund rate, p95 checkout latency.
Randomize by user_id. MDE: +1.5pp absolute on a 12% baseline."
Step 2 — Sample size from MDE, baseline, and power
Defaults: power 0.80, α 0.05 (two-sided). The MDE is yours to choose — it is the smallest effect that would actually change what you do.
Rule: required n scales with ~1/MDE². Why: halving the smallest effect you care to detect roughly quadruples the traffic and time. This is the single most expensive decision in the design, so set the MDE to a business threshold, never to "whatever is small."
For a conversion rate (proportion):
from statsmodels.stats.power import NormalIndPower
from statsmodels.stats.proportion import proportion_effectsize
p1, p2 = 0.12, 0.135 # baseline, baseline + MDE (1.5pp)
h = proportion_effectsize(p1, p2) # Cohen's h (arcsine transform)
n = NormalIndPower().solve_power(effect_size=h, alpha=0.05, power=0.80, ratio=1.0)
print(int(-(-n // 1))) # n PER ARM, rounded up
For a continuous metric (revenue per user, time on page) use Welch-style sizing:
from statsmodels.stats.power import TTestIndPower
effect = mde_in_units / pooled_std # Cohen's d
n = TTestIndPower().solve_power(effect_size=effect, alpha=0.05, power=0.80, ratio=1.0)
Then convert n to a calendar plan: days = ceil((n_per_arm * num_arms) / daily_eligible_users). If that
is 9 days, run a clean two full weeks anyway — weekday/weekend mix is part of the population, and a
6-day test oversamples whoever shows up Tuesday. Full worked example (12% baseline, +1.5pp MDE, 80%
power) plus runnable sizing, n→duration, CUPED θ and SRM snippets: references/sample-size-and-cuped.md.
Step 3 — Run discipline
Fixed horizon is the default. Commit to the n/date from Step 2 and read the result once, at the end.
Do not peek and stop at first significance. Why: checking repeatedly and stopping the moment p < 0.05 inflates the Type-I error far above 5% — with enough looks, a null test crosses 0.05 most of the time. If you genuinely need to stop early, use a sequential / always-valid method (confidence sequences, e.g. Netflix's anytime-valid CIs) that holds Type-I error under continuous monitoring. Sequential is strong for killing losers early and weak for calling winners early — for a confident win, the fixed-horizon read is tighter.
Gate on SRM before you trust anything. Compute a chi-square test on the observed split versus the intended ratio. If p < 0.001 the assignment or logging is broken — a bot filter dropping one arm, a redirect, a caching bug. Fix the instrumentation and rerun; do not "adjust for it."
The peeking Type-I math, sequential/always-valid options, SRM diagnosis, novelty/primacy effects,
Simpson's paradox in segments and HARKing all live in references/pitfalls.md.
Step 4 — Analyze
Pick the test by metric type:
| Metric type | Test |
|---|---|
| Binary conversion (proportion) | Two-proportion z-test (statsmodels.stats.proportion.proportions_ztest) |
| Continuous, roughly normal / large n | Welch's t-test (`scipy.stats.ttest_ind(..., equal_v |