跳至正文
OpenAgentSkill
浏览已开放 · 目录记录日期 2026-11-21 · 自动更新和账户服务暂时关闭。
开发与测试

ab-test-setup-global

Use when the user wants a VALID experiment instead of a guess — hypothesis, one variable, sample size and runtime math, statistical significance, primary versus secondary metrics, multi-arm designs, and a results template, across Optimizely, VWO, and native Meta and Google tests. Trigger on 'A/B test', 'split test', 'how long should I run the test', 'is this result significant', 'test two versions', 'which creative is actually better'. Also use when a winner was declared after two days on tiny numbers. Not for — scaling the proven winner, see `scaling-ads-global`; analyzing data already collected, see `data-analysis-global`; auditing the account, see `ads-audit-global`.

huytran1120 · 已有静态检查记录

安装与使用

运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。

npx skills add huytran1120/vie-agentic-skills --skill ab-test-setup-global

技能说明


name: ab-test-setup-global description: "Use when the user wants a VALID experiment instead of a guess — hypothesis, one variable, sample size and runtime math, statistical significance, primary versus secondary metrics, multi-arm designs, and a results template, across Optimizely, VWO, and native Meta and Google tests. Trigger on 'A/B test', 'split test', 'how long should I run the test', 'is this result significant', 'test two versions', 'which creative is actually better'. Also use when a winner was declared after two days on tiny numbers. Not for — scaling the proven winner, see scaling-ads-global; analyzing data already collected, see data-analysis-global; auditing the account, see ads-audit-global." metadata: version: 1.0.1 category: performance language: en license: MIT triggers:

  • "A/B test"
  • "split test"
  • "multivariate test"
  • "experiment design"
  • "statistical significance"
  • "sample size calculator" output: A .md file containing hypothesis, sample size calculation, primary/secondary metrics, test setup, timeline, and a results template ready for analysis related:
  • product-marketing-context-global
  • data-analysis-global
  • performance-eval-global
  • ads-audit-global

A/B Test Setup (Global)

Run experiments that produce decisions, not noise. Most "A/B tests" in marketing are underpowered, peeked-at, and badly hypothesized — meaning the team learns nothing and ships the louder variant.


For Newbies

A valid A/B test answers one question: "Did this change cause a real improvement, or am I seeing noise?"

To answer it credibly you need four things:

  1. A specific hypothesis with a numeric prediction
  2. One variable changed (everything else identical)
  3. Enough sample to detect the effect you care about
  4. Statistical significance before you call a winner (typically p < 0.05)

If any one of these is missing, you don't have an A/B test — you have a coin flip with extra steps.

Common newbie mistake: running a test for 3 days, seeing variant B 40% higher, declaring victory, and shipping. Three days is too short to absorb day-of-week effects, and small samples produce wild swings. Variant B may revert (or reverse) by day 14.


Step 0 — Read Context

Read .agents/product-marketing-context.md if it exists. Audience size, average traffic, and current conversion rate determine whether a test is even feasible.


Step 1 — Information Gathering

Ask up to 4 questions:

  1. What are you testing? (Ad headline / Landing page section / Email subject / Pricing display / CTA button / Creative video)
  2. Primary metric? (CTR / Conversion rate / CPM / CPA / Revenue / Open rate / Reply rate)
  3. Daily traffic to the test surface? (Needed for sample size and duration)
  4. Goal of the test? (Lift X% on primary metric / Pick a winner among N candidates / Validate a strategic hypothesis)

The 7 Principles of a Valid A/B Test

1. Test exactly one variable

The cardinal rule. Change two things at once and you cannot attribute the result.

  • Bad: "I changed the headline, the hero image, and the CTA color." → You learn nothing about which element drove the lift.
  • Good: Change only the headline. Image, CTA, layout, traffic source, and audience targeting are identical.

If you must test multiple changes, use a multivariate test (MVT) — but those need much more traffic (often 4×–8× a single A/B).

2. Hypothesize with a number

Format: "If we [change X], [metric Y] will increase by [Z%] because [reason]."

  • Good: "If we change the CTA from 'Sign up' to 'Get my free demo,' conversion rate will increase by 15% because action-specific language reduces ambiguity."
  • Bad: "The new copy will be better." (No metric, no number, no causal reasoning — un-testable.)

The "because" matters: if your hypothesis is wrong but the reasoning was sound, you've still learned something generalizable.

3. Sufficient sample size

Don't stop early. Statistical tests need adequate data to distinguish signal from noise.

  • Minimum rule of thumb: 100 conversions per variant (not 100 visitors)
  • Better: Calculate sample size up front based on baseline conversion rate and minimum detectable effect (formula below)

4. Sufficient duration

Run for whole weeks, not 3 days, not 10 days. Different weekdays produce different audience behavior — Monday B2B traffic is not Saturday DTC traffic.

  • Minimum: 7 days
  • Recommended: 14 days
  • Watch for: holidays, paydays, monthly billing cycles, ad spend ramp-ups

5. Don't peek

Looking at results every hour and stopping when "B looks good" is the most common error in marketing experimentation. Early peeks combined with early stops dramatically inflate false positive rates.

  • Define the end date in advance. Honor it.
  • If you must monitor, use sequential testing methods designed for it (Bayesian frameworks like Optimizely's Stats Engine, or platforms with built-in sequential controls).

6. Statistical significance: p < 0.05

Most marketing teams use 95% confidence (p-value < 0.05) as the bar.

  • p-value < 0.05 → less than 5% chance the observed difference is random
  • p-value 0.05–0.10 → suggestive but inconclusive — extend the test
  • p-value > 0.10 → no evidence of an effect — keep control or test something else

For high-stakes tests (pricing, branding) consider 99% confidence (p < 0.01).

7. Document everything

Write down:

  • Hypothesis (with number)
  • Start date / end date
  • Sample size achieved
  • Primary metric, secondary metrics
  • Result + p-value
  • Decision + reasoning
  • What you'd test next

A documented test history prevents your team from re-testing things that already failed and from forgetting why you made past decisions.


Sample Size Calculation

Quick formula

Sample size per variant ≈ 16 × p × (1 − p) / MDE²

where:
  p   = baseline conversion rate (e.g. 0.03 = 3%)
  MDE = minimum de
阅读完整来源 ↗