ab-testing
When the user wants to plan, design, or implement an A/B test or experiment, or build a growth experimentation program.
huytran1120 · 已有静态检查记录
安装与使用
运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。
npx skills add huytran1120/vie-agentic-skills --skill ab-testing技能说明
name: ab-testing description: "When the user wants to plan, design, or implement an A/B test or experiment, or build a growth experimentation program." risk: critical source: https://github.com/coreyhaines31/marketingskills/tree/main/skills/ab-testing source_repo: coreyhaines31/marketingskills source_type: community date_added: 2026-07-01 license: MIT license_source: https://github.com/coreyhaines31/marketingskills/blob/main/LICENSE
A/B Test Setup
When to Use
Use to plan, design or implement a controlled experiment or experimentation program.
You are an expert in experimentation and A/B testing. Your goal is to help design tests that produce statistically valid, actionable results.
Initial Assessment
Check for product marketing context first:
If .agents/product-marketing.md exists (or .claude/product-marketing.md, or the legacy product-marketing-context.md filename, in older setups), read it before asking questions. Use that context and only ask for information not already covered or specific to this task.
Before designing a test, understand:
- Test Context - What are you trying to improve? What change are you considering?
- Current State - Baseline conversion rate? Current traffic volume?
- Constraints - Technical complexity? Timeline? Tools available?
Core Principles
1. Start with a Hypothesis
- Not just "let's see what happens"
- Specific prediction of outcome
- Based on reasoning or data
2. Test One Thing
- A/B tests may compare a bundle, but then estimate the bundle's overall effect; individual component effects require a suitable factorial design
3. Statistical Rigor
- Pre-determine sample size
- Don't peek and stop early
- Commit to the methodology
4. Measure What Matters
- Primary metric tied to business value
- Secondary metrics for context
- Guardrail metrics to prevent harm
Hypothesis Framework
Structure
Because [observation/data],
we believe [change]
will cause [expected outcome]
for [audience].
We'll know this is true when [metrics].
Example
Weak: "Changing the button color might increase clicks."
Strong: "Because users report difficulty finding the CTA (per heatmaps and feedback), we believe making the button larger and using contrasting color will increase CTA clicks by 15%+ for new visitors. We'll measure click-through rate from page view to signup start."
Test Types
| Type | Description | Traffic Needed |
|---|---|---|
| A/B | Two versions, single change | Moderate |
| A/B/n | Multiple variants | Higher |
| MVT | Multiple changes in combinations | Very high |
| Split URL | Different URLs for variants | Moderate |
Sample Size
Quick Reference
The root and reference tables previously gave inconsistent values. Calculate using a named method with recorded baseline, MDE units, alpha, power, sidedness, allocation and multiple-comparison adjustment; see the reference guide. Do not use unbound numeric shortcuts.
Calculators:
For detailed sample size tables and duration calculations: See references/sample-size-guide.md
Metrics Selection
Primary Metric
- Single metric that matters most
- Directly tied to hypothesis
- What you'll use to call the test
Secondary Metrics
- Support primary metric interpretation
- Explain why/how the change worked
Guardrail Metrics
- Things that shouldn't get worse
- Stop test if significantly negative
Example: Pricing Page Test
- Primary: Plan selection rate
- Secondary: Time on page, plan distribution
- Guardrail: Support tickets, refund rate
Designing Variants
What to Vary
| Category | Examples |
|---|---|
| Headlines/Copy | Message angle, value prop, specificity, tone |
| Visual Design | Layout, color, images, hierarchy |
| CTA | Button copy, size, placement, number |
| Content | Information included, order, amount, social proof |
Best Practices
- Single, meaningful change
- Bold enough to make a difference
- True to the hypothesis
Traffic Allocation
| Approach | Split | When to Use |
|---|---|---|
| Standard | 50/50 | Default for A/B |
| Conservative | 90/10, 80/20 | Limit risk of bad variant |
| Ramping | Start small, increase | Technical risk mitigation |
Considerations:
- Consistency: Users see same variant on return
- Balanced exposure across time of day/week
Assignment and safety checks
Define stable randomization unit, eligibility/exposure, conversion window and intention-to-treat analysis before launch. Check sample-ratio mismatch, missing events, bot/internal traffic, contamination across devices and repeated-user dependence; broken instrumentation can invalidate significance. Predeclare segments/comparisons and separate exploratory findings. Monitor harmful technical/guardrail effects with a planned stop policy even during fixed-horizon tests. Sequential methods require their own valid stopping rules. Every variant must preserve accessibility, honest offers, consent/privacy and security. Testing does not authorize unrequested production rollout or data sharing.
Implementation
Client-Side
- JavaScript modifies page after load
- Quick to implement, can cause flicker
- Tools: PostHog, Optimizely, VWO
Server-Side
- Variant determined before render
- No flicker, requires dev work
- Tools: PostHog, LaunchDarkly, Split
Running the Test
Pre-Launch Checklist
- Hypothesis documented
- Primary metric defined
- Sample size calculated
- Variants implemented correctly
- Tracking verified
- QA completed on all variants
During the Test
DO:
- Monitor for technical issues
- Check segment quality
- Document external factors
**Avoi
阅读完整来源 ↗