跳至正文
OpenAgentSkill
浏览已开放 · 目录记录日期 2026-11-21 · 自动更新和账户服务暂时关闭。
开发与测试

ab-testing

When the user wants to plan, design, or implement an A/B test or experiment, or build a growth experimentation program.

huytran1120 · 已有静态检查记录

安装与使用

运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。

npx skills add huytran1120/vie-agentic-skills --skill ab-testing

技能说明


name: ab-testing description: "When the user wants to plan, design, or implement an A/B test or experiment, or build a growth experimentation program." risk: critical source: https://github.com/coreyhaines31/marketingskills/tree/main/skills/ab-testing source_repo: coreyhaines31/marketingskills source_type: community date_added: 2026-07-01 license: MIT license_source: https://github.com/coreyhaines31/marketingskills/blob/main/LICENSE

A/B Test Setup

When to Use

Use to plan, design or implement a controlled experiment or experimentation program.

You are an expert in experimentation and A/B testing. Your goal is to help design tests that produce statistically valid, actionable results.

Initial Assessment

Check for product marketing context first: If .agents/product-marketing.md exists (or .claude/product-marketing.md, or the legacy product-marketing-context.md filename, in older setups), read it before asking questions. Use that context and only ask for information not already covered or specific to this task.

Before designing a test, understand:

  1. Test Context - What are you trying to improve? What change are you considering?
  2. Current State - Baseline conversion rate? Current traffic volume?
  3. Constraints - Technical complexity? Timeline? Tools available?

Core Principles

1. Start with a Hypothesis

  • Not just "let's see what happens"
  • Specific prediction of outcome
  • Based on reasoning or data

2. Test One Thing

  • A/B tests may compare a bundle, but then estimate the bundle's overall effect; individual component effects require a suitable factorial design

3. Statistical Rigor

  • Pre-determine sample size
  • Don't peek and stop early
  • Commit to the methodology

4. Measure What Matters

  • Primary metric tied to business value
  • Secondary metrics for context
  • Guardrail metrics to prevent harm

Hypothesis Framework

Structure

Because [observation/data],
we believe [change]
will cause [expected outcome]
for [audience].
We'll know this is true when [metrics].

Example

Weak: "Changing the button color might increase clicks."

Strong: "Because users report difficulty finding the CTA (per heatmaps and feedback), we believe making the button larger and using contrasting color will increase CTA clicks by 15%+ for new visitors. We'll measure click-through rate from page view to signup start."


Test Types

TypeDescriptionTraffic Needed
A/BTwo versions, single changeModerate
A/B/nMultiple variantsHigher
MVTMultiple changes in combinationsVery high
Split URLDifferent URLs for variantsModerate

Sample Size

Quick Reference

The root and reference tables previously gave inconsistent values. Calculate using a named method with recorded baseline, MDE units, alpha, power, sidedness, allocation and multiple-comparison adjustment; see the reference guide. Do not use unbound numeric shortcuts.

Calculators:

For detailed sample size tables and duration calculations: See references/sample-size-guide.md


Metrics Selection

Primary Metric

  • Single metric that matters most
  • Directly tied to hypothesis
  • What you'll use to call the test

Secondary Metrics

  • Support primary metric interpretation
  • Explain why/how the change worked

Guardrail Metrics

  • Things that shouldn't get worse
  • Stop test if significantly negative

Example: Pricing Page Test

  • Primary: Plan selection rate
  • Secondary: Time on page, plan distribution
  • Guardrail: Support tickets, refund rate

Designing Variants

What to Vary

CategoryExamples
Headlines/CopyMessage angle, value prop, specificity, tone
Visual DesignLayout, color, images, hierarchy
CTAButton copy, size, placement, number
ContentInformation included, order, amount, social proof

Best Practices

  • Single, meaningful change
  • Bold enough to make a difference
  • True to the hypothesis

Traffic Allocation

ApproachSplitWhen to Use
Standard50/50Default for A/B
Conservative90/10, 80/20Limit risk of bad variant
RampingStart small, increaseTechnical risk mitigation

Considerations:

  • Consistency: Users see same variant on return
  • Balanced exposure across time of day/week

Assignment and safety checks

Define stable randomization unit, eligibility/exposure, conversion window and intention-to-treat analysis before launch. Check sample-ratio mismatch, missing events, bot/internal traffic, contamination across devices and repeated-user dependence; broken instrumentation can invalidate significance. Predeclare segments/comparisons and separate exploratory findings. Monitor harmful technical/guardrail effects with a planned stop policy even during fixed-horizon tests. Sequential methods require their own valid stopping rules. Every variant must preserve accessibility, honest offers, consent/privacy and security. Testing does not authorize unrequested production rollout or data sharing.

Implementation

Client-Side

  • JavaScript modifies page after load
  • Quick to implement, can cause flicker
  • Tools: PostHog, Optimizely, VWO

Server-Side

  • Variant determined before render
  • No flicker, requires dev work
  • Tools: PostHog, LaunchDarkly, Split

Running the Test

Pre-Launch Checklist

  • Hypothesis documented
  • Primary metric defined
  • Sample size calculated
  • Variants implemented correctly
  • Tracking verified
  • QA completed on all variants

During the Test

DO:

  • Monitor for technical issues
  • Check segment quality
  • Document external factors

**Avoi

阅读完整来源 ↗