power-analysis
Use when the user is planning a two-arm comparison (an A/B test, a simple RCT, a behavioral study, or a two-model/two-config evaluation) and needs to size it and preregister it before collecting data — finding the per-group sample size that hits target statistical power for the smallest effect worth detecting, auditing the design against a validity checklist, and locking it in a preregistration. Only for a single two-arm comparison with one primary outcome. Not for factorial, repeated-measures, clustered/multilevel, time-series, adaptive, or survival designs; not for analyzing data already collected; not for choosing the outcome or manipulation from domain knowledge.
gaasher · AI review on file
Installation
Read the source and dependencies before running a command. Copying does not install or execute anything.
npx skills add gaasher/Agent-Loop-Skills --skill power-analysisAbout this skill
name: power-analysis description: > Use when the user is planning a two-arm comparison (an A/B test, a simple RCT, a behavioral study, or a two-model/two-config evaluation) and needs to size it and preregister it before collecting data — finding the per-group sample size that hits target statistical power for the smallest effect worth detecting, auditing the design against a validity checklist, and locking it in a preregistration. Only for a single two-arm comparison with one primary outcome. Not for factorial, repeated-measures, clustered/multilevel, time-series, adaptive, or survival designs; not for analyzing data already collected; not for choosing the outcome or manipulation from domain knowledge. compatibility: Requires Python 3.9+ metadata: version: "0.1.0"
Power Analysis Loop
A power-analysis-and-preregister loop for a two-arm comparison. The artifact is the study's statistical plan; the feedback signal is two parts — statistical power (estimated by Monte-Carlo simulation of the planned test) and a count of validity flaws. Each iteration simulates power, solves for the sample size that reaches the target, audits the design for flaws, and revises — until power clears the target and the flaw list is empty. The deliverable is a sample-size justification plus a preregistration that pins the hypothesis, primary outcome, analysis, sample size, and stopping rule before any data is seen.
Scope & limitations
This loop does exactly three things, in a loop: (1) computes power and required sample size for a
two-group comparison by simulation, (2) runs a fixed validity checklist over the design,
and (3) writes a preregistration. The vendored power model (tools/power_sim.py) covers
two-sample mean (continuous outcome) and two-proportion (binary outcome) tests only.
It is not a general experiment designer. It does not handle factorial, repeated-measures, clustered/multilevel, time-series, adaptive, or survival designs; it does not pick your outcome measure or manipulation from domain knowledge; and it does not analyze data you have already collected. For those, the power numbers here do not apply — use a design-appropriate power method. If the study is not a simple two-arm comparison, say so and stop rather than reporting a power that does not match the planned analysis.
When to use
Use this to size and preregister one two-arm comparison whose primary outcome is a continuous mean or a binary rate. Default to powering for the minimal effect of interest the user states; if they are unsure of that effect, help them set it from a baseline and a smallest-meaningful difference rather than an optimistic guess — a design "powered" for an effect bigger than reality is a fiction. If the study is not a two-arm comparison, stop and point to a design-appropriate method.
Setup
Resolve bindings interactively. If loop.run.yaml exists in the working dir, load it, confirm the
values in one line, and skip to the loop. Otherwise: on Claude Code (the AskUserQuestion tool is
available) infer a likely value for each binding and present it as the recommended option; on other
hosts ask each as a quoted plain-text prompt. Then write loop.run.yaml (format:
examples/run.example.yaml) and confirm the values before creating any other files.
| binding | meaning | default | how to infer |
|---|---|---|---|
<hypothesis> | the claim the experiment tests | — | ask the user |
<outcome> | primary outcome type + minimal effect of interest: continuous (baseline_mean, sd, min_effect) or binary (baseline_rate, min_lift) | — | ask; this fixes the effect size power is computed at |
<target_power> | power the design must clear | 0.80 | — |
<alpha> | significance level | 0.05 | — |
<power_cmd> | invocation of the vendored simulator | python3 <skill_dir>/tools/power_sim.py --design <two-sample-mean|two-proportion> --effect <e> [--sd <sd> | --baseline <p0>] --alpha <alpha> --n <n_per_group> | — |
<design_doc> | output design + preregistration file | <sandbox_root>/design.md | — |
<sandbox_root> | where design + ledger live | ./sandbox | — |
<budget> | max iterations | 8 | — |
<power_cmd> prints one JSON object, {"power", "n_per_group", ...}. Run it to get the power; never
estimate power by hand.
The loop
Copy this checklist and tick items off:
- Iteration 0 — draft the design to
<design_doc>; record nothing as final. - Simulate power: run
<power_cmd>at the currentnand the assumed effect. - Solve N: if power
< <target_power>, re-run at largern(step up, then bisect) until it clears. - Audit validity: list every flaw from the checklist below.
- Revise: fix the highest-priority flaw, set
nto the power-adequate value, update<design_doc>(+ Preregistration section). - Append a ledger row; stop when power clears the target and no flaws remain, or at
<budget>.
Iteration 0 — draft. Write a first design to <design_doc>: the arms/conditions, the unit of
analysis and how units are assigned, the primary outcome and the exact planned test, the assumed effect
size (from <outcome>), and a first sample-size guess. Record nothing as final yet.
Then, until stop (power met + no flaws, or budget):
- Simulate power. Run
<power_cmd>at the current per-groupnand the assumed effect, with the--designmatching the planned test. Record the achieved power. - Solve N. If
power < <target_power>, re-run the simulation at largern— step up (e.g. double), then bisect — until power clears the target, and adopt thatn. - Audit validity. Check the design against the checklist and list every flaw found:
- Confounding / no control — is there a concurrent control group, or is the comparison against a historical/other-source baseline that differs in other ways?