controls-samples-diagnostics
Diagnostics for controls, samples, and denominators that define the estimand: mediator controls, control horse races, same-source covariates, covariate-by-period under fixed effects, treated shares from the regression sample, and denominator transportability. Invoke from `code-and-data` whenever a horse race, mediator, attribution claim, subset restriction, or same-source covariate touches a target result.
isitcredible · Static checks on file
Installation
Read the source and dependencies before running a command. Copying does not install or execute anything.
npx skills add isitcredible/research_skills --skill controls-samples-diagnosticsAbout this skill
name: controls-samples-diagnostics
description: >-
Diagnostics for controls, samples, and denominators that define the
estimand: mediator controls, control horse races, same-source covariates,
covariate-by-period under fixed effects, treated shares from the
regression sample, and denominator transportability. Invoke from
code-and-data whenever a horse race, mediator, attribution claim,
subset restriction, or same-source covariate touches a target result.
Controls, samples, and denominators diagnostics
Establish what each control, sample restriction, and denominator does to the estimand before reading any coefficient movement. Controls are treated as free insurance, but a control can redefine the question or inject bias outright: control for the mechanism and the effect "disappears"; control for a variable built from the same source as the regressor and the two fight over shared measurement error; restrict the sample and the claim quietly becomes a claim about the subset; or add a control that is itself a collider or a near-instrument and manufacture the bias it appears to prevent (Cinelli, Forney & Pearl 2024). A coefficient drop can mean confounding exposed — or the estimand changed mid-argument, or a bad control was added; the checks below tell them apart.
Trigger: a control horse race, a mediator control, an aggregate attribution
claim, a subset restriction, covariates built from the same source as
the regressor, a control the authors added to demonstrate robustness (a
kitchen-sink covariate set), a pre-treatment control claimed as safe, a
control or sample plausibly affected by the treatment or outcome, or
any discretionary specification choice not forced by
theory — a single-unit/small-group dummy, an ad-hoc interaction, a bespoke
sample carve-out, a hand-picked control, an unruled tuning constant — in
any target spec. NOT_APPLICABLE when none of these touches a target result.
Method crib (interim — WP-audited):
../../references/methods/goldsmithpinkham_hull_kolesar_2024_contamination.md —
contamination bias under multiple treatments + controls; published AER
version pending.
Core checks
-
Coefficient movement versus standard-error inflation. Print before/after estimates, intervals, SEs, sample, and identifying variation; report changes in units (percentage changes only with a nonzero, meaningful denominator). A stable coefficient with a wider interval indicates reduced precision; a coefficient moving toward zero indicates attenuation in that specification. Neither pattern establishes whether the added control is valid. Check temporal and causal ordering, collider/mediator risks, changed estimands, and loss of identifying variation. VIF diagnoses linear dependence among regressors, not causal validity: low VIF cannot prove confounding or a zero causal effect. Interpret the result against the paper's actual claim and a substantively justified specification. Two naming requirements make the check answerable: for every contested control, name which catalog model the paper's causal structure implies (the Cinelli–Forney–Pearl crib's model number) — if the paper has no causal model, that absence is itself the finding — and name the reference specification against which movement is measured, with the reason; movement relative to an unnamed baseline supports opposite stories. When the paper's defense is coefficient stability across control sets, route to the OVB-sensitivity record (Cinelli–Hazlett robustness value benchmarked to named covariates, or Oster δ/Rmax) — "the coefficient was stable" alone is not a sensitivity analysis. And test whether the stability check could have detected anything: when the added controls are noisy proxies for the true confounder, attenuation makes coefficient invariance expected even under confounding — the more powerful test is the balancing regression (proxy on the left, treatment on the right), per
../../references/methods/pei_pischke_schwandt_2019_left_right.md. A paper resting identification on RHS stability with poorly measured proxies has weak evidence, whatever the stability. -
Mediator controls identified and removed. A control that sits on the causal path from treatment to outcome answers a different question; re-estimate without it and report both numbers with the distinction stated. Before prescribing removal, ask whether the mediator is itself confounded (a treatment→mediator confounder or a mediator↔outcome confounder): then both the with- and without-control estimates are biased and the finding is "unidentified without stronger assumptions," not "the mediator inflated it."
-
Bad controls that inject bias — test by removal, watch the direction. Beyond mediators, a control can create the bias it appears to remove. Re-estimate without each suspect control and report not just whether significance moves but which way the point estimate moves: movement toward the null or other anchors is sensitivity evidence, not proof the control was harmful. Establish control validity from the causal structure and provenance; the null and other estimates are not known truth.
- M-bias (pre-treatment collider). Do not exempt a covariate because it is "pre-treatment." If it is a common effect of unobserved causes of both treatment and outcome, conditioning on it opens a back-door; test removal and interpret movement conditional on the proposed causal structure.
- Bias amplification (near-instrument). Flag any covariate that predicts the treatment strongly but has no independent path to the outcome. Under residual confounding it amplifies existing bias; remove it and report the move.
- Selection / case-control (collider or descendant of the outcome). Treat a post-outcome control, or a sample/frame selected on a variable the outcome affects, as a collider until shown otherwise — it induces a spurious association that no coefficient st