experiment-diagnostics
Diagnostics for randomized experiments: assignment integrity, attrition and missingness, intent-to-treat estimation, prespecified outcome families with multiplicity control, randomization inference, and noncompliance handling. Invoke from `code-and-data` whenever a target result rests on randomized assignment.
isitcredible · Static checks on file
Installation
Read the source and dependencies before running a command. Copying does not install or execute anything.
npx skills add isitcredible/research_skills --skill experiment-diagnosticsAbout this skill
name: experiment-diagnostics
description: >-
Diagnostics for randomized experiments: assignment integrity, attrition
and missingness, intent-to-treat estimation, prespecified outcome families
with multiplicity control, randomization inference, and noncompliance
handling. Invoke from code-and-data whenever a target result rests on
randomized assignment.
Experiment diagnostics
Randomization buys identification at baseline and nothing after it. Check everything that happens after: attrition, noncompliance, outcome multiplicity, and analysis-unit mismatches can each quietly rebuild the selection problems randomization removed, and each has its own test.
Trigger: randomized assignment behind any target result. NOT_APPLICABLE
for observational designs.
Method cribs (source-audited):
../../references/methods/young_2019_channelling_fisher.md and
../../references/methods/morgan_rubin_2012_rerandomization.md —
restricted-assignment-set RI; a screened/rerandomized design analyzed
with pure-randomization theory gets conservative inference (safe
direction, but still the wrong distribution).
Core checks
- Assignment and analysis-unit match. Randomized at the clinic,
analyzed at the patient, is a clustering error; inference belongs at the
level of assignment, with the cluster count reported. The whether-to-
cluster decision is a design question, not a residual-correlation test
(
../../references/methods/abadie_athey_imbens_wooldridge_2023_clustering.md): cluster-level assignment is what forces the cluster adjustment, and in a fully unit-randomized trial clustering the SEs is a costly choice, not a default — check which variance component the design actually carries. - Collapse to the assignment unit (point estimate). The unit-match check
fixes inference; when treatment is assigned at a coarser level than the
analysis rows, also collapse to one row per assigned unit (average the
replicate subsamples) and confirm the coefficient itself, not only its
interval, is read at that resolution. See
../../references/design-matrix.md, "Collapse-test record". - Assignment integrity. Balance on predetermined covariates, and the randomization procedure as actually implemented — batch, stratified, re-randomized — reflected in the analysis.
- Post-treatment conditioning. Check whether the headline spec controls
for, or subsets on, a post-randomization variable (a mediator, an
endogenous screen, an adjustment set chosen on results). An experiment
re-imports bias exactly here; where present, hand to
controls-samples-diagnosticswith the model named. - Attrition and missingness. Overall and differential by arm; when the differential is non-trivial (say, the arms' attrition rates differ beyond a stated tolerance, or attrition correlates with baseline outcomes), run Lee bounds or a stated worst-case analysis. A differential that is not statistically significant is not clearance: the bounds show whether it matters beside the effect. Attrition is the experiment's selection problem readmitted through the back door.
- Intent-to-treat estimate. The ITT is the design's protected quantity; report it before any per-protocol or treatment-on-treated number.
- Prespecified outcome family and multiplicity adjustment. Compare the
outcomes reported against the outcomes collected (registry, pre-analysis
plan, survey instrument), define the family from the claim structure, and
adjust. An experiment with twenty outcomes and one star is the
multiplicity failure in its purest form. Where a pre-analysis plan or
registry entry exists, cite
../../references/methods/olken_2015_preanalysis.mdfor the credibility limits of pre-registration; where the multiplicity is visible only in the paper's own spec table, report the pile-up of p-values just below the threshold descriptively — correlated specifications cluster around the baseline by construction, so the pile-up is not by itself evidence of search; price it with the specification sweep — and do not run the p-curve tests of../../references/methods/elliott_kudrin_wuthrich_2022_phacking.mdon one paper's table: they are across-study tests. (Humps at cutoffs are what non-increasingness tests detect; the "no power" case is p-hacking that shifts mass left without humps, which the paper's bounds tests target.) - Randomization inference where feasible — the design's own exact test, using the actual assignment mechanism. Name the permuted statistic: the t-statistic permutation under the true mechanism is the standard object (Young's randomization-t). Coefficient permutations (the randomization-c) test the same sharp null, but prefer the -t: it stays exact in joint tests and robust under heterogeneous effects, where the -c does not (Young 2019, §III.B). The permutation must respect the mechanism — permute within strata for a stratified design, condition on the re-randomization criterion where the design used one — or the RI p-value is invalid. Permuting arm labels while leaving arm-derived columns (treatment dummies, control flags, cell memberships) at their observed values is not a permutation of the assignment — it permutes only the margins whose columns are actually re-derived, and an "invariance" or degeneracy rate measured under it is a harness artifact. Re-derive every assignment-descended column from the permuted labels, and validate the generator before trusting it: rebuilding from the observed labels must reproduce the shipped columns exactly. The per-coefficient test inherits the omnibus caveat: it is a sharp-null test. Where RI is infeasible, record why and what was run instead. A degenerate or invariant direct statistic is NOT infeasibility — when the natural statistic is mechanically invariant under valid permutations (a pooled or selected cell whose membership barely moves), the obliga