small-n-diagnostics
Diagnostics for small samples and minimum detectable effects: MDE at conventional power against claimed and plausible effect sizes, lean re-tests when saturated controls overfit, and estimator-specific small-sample artifacts. Invoke from `code-and-data` whenever units or clusters are few, or a null carries interpretive weight.
isitcredible · 已有静态检查记录
安装与使用
运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。
npx skills add isitcredible/research_skills --skill small-n-diagnostics技能说明
name: small-n-diagnostics
description: >-
Diagnostics for small samples and minimum detectable effects: MDE at
conventional power against claimed and plausible effect sizes, lean
re-tests when saturated controls overfit, and estimator-specific
small-sample artifacts. Invoke from code-and-data whenever units or
clusters are few, or a null carries interpretive weight.
Small-N diagnostics
Establish what the design could actually detect and what its reported precision is worth. A small sample cuts in both directions at once: too little power to detect a plausible effect, and too few degrees of freedom to survive the controls piled on it — so a null proves nothing on its own, and a precise-looking estimate may owe its precision to a small-sample artifact of the estimator. The checks below put numbers on both directions so the result can be read at its true weight.
Trigger: few units or few clusters behind any target result, or a null or
near-null that carries interpretive weight — whether the paper leans on the
null or the battery produced it. NOT_APPLICABLE when the design has ample
units and no null is load-bearing.
Core checks
-
Minimum detectable effect at conventional power, computed against both the claimed effect size and the effects the mechanism or the specialist literature makes plausible. State the comparison in the same units as the claim: "this design could detect an effect of X; the plausible effect is Y." Underpowered does not mean refuted — and a null from an underpowered design is uninformative in both directions — the evidence
claimsneeds to decide whether a null is informative. -
Lean and bivariate re-tests — when saturated controls sit on a small sample: re-estimate with minimal controls. In a small sample, a control set approaching the number of observations manufactures fit; the lean specification shows what the data alone support.
-
Effective number of clusters, not nominal. For every clustered target compute the partial-leverage effective count G* (Carter et al. 2017; ρ = 0 iff the spec carries cluster FEs) — see
../../references/methods/mackinnon_nielsen_webb_2023_few_clusters.md. The prior question — whether and at what level clustering is owed — is a design question, not a residual-correlation test: it is answered from how clusters entered the sample and how treatment was assigned (../../references/methods/abadie_athey_imbens_wooldridge_2023_clustering.md); a design can fail at either stage — cluster at the right level and still have G* ≈ 2, or cluster on residual correlation alone where the design carries no cluster-level assignment. A shift-share regressor is a clustered target at the shock level whatever the unit-level SEs are. Heteroskedasticity-robust SEs over units that look independent do not exempt it, because the units share the shocks. Run this check there, through the shift-share module's shock-level jackknife row. A few-cluster tool applied to unit-level cells (a wild bootstrap over demographic or regional cells) is not the shock-level answer, and must not be reported as one. Fifty nominal clusters with one dominant leverage behave as ~1–2.THE WILD CLUSTER BOOTSTRAP IS NOT THE FEW-CLUSTERS ANSWER. Your training says it is, and that plain
boottestis how to run it. Both are wrong when G* is small. Plainboottestis WCR-C. With one dominant cluster its draws mostly re-sign that cluster, and its p-value can be far too small or far too large. Runs have repeatedly headlined WCR-C p-values that the jackknife and randomization overturned.../../scripts/check_inference.pyfails the run on it. When G* is small or leverages are unequal, do this, in this order:- G* and partial leverages, top clusters named.
- CV3 (delete-one-cluster jackknife) leads, beside CV1. Re-fit non-cluster FEs
and controls in each leave-out; partial out only FEs nested within the clusters.
When coding CV3 by hand (because
summclustis unavailable or the specification uses multi-way FEs or weights), re-estimate the absorbed FEs and controls inside every leave-one-cluster-out iteration — never demean or partial out non-cluster FEs once on the full sample before the loop, which leaks the omitted cluster into the fixed effects and corrupts the leave-out variance. The jackknife emits a variance, which the battery carries downstream: SE ratios against the paper's CV1, VtF/tF inputs, the quantify-inference rule. Its β̂^(g) leave-out distribution shows which cluster carries the estimate. The CV1→CV3 gap is the leverage finding. - Restricted wild bootstrap, second, led by WCR-S (
boottest …, jk nocifollowed immediately byassert !missing(r(p)); R/Pythonbootstrap_type = "31").boottestprints a confidence set by default for a single hypothesis, and itsjkresidual adds an(M_{gg}^{-1} − I)X_g t₁term that vanishes only at a null of zero, sonociis mandatory (check_inference.pyF9) and any non-zero null or confidence/Anderson–Rubin set must be inverted by transforming the outcome (y − b₀·xory − b₀·endog) and testing zero over a grid (boottest …, arruns an unrestricted bootstrap DGP and failscheck_inference.pyF8 unless taggedAR-boottest-robustnessas a companion row). Becauseboottestexits with_rc = 0andr(p) = .when it refuses a model (e.g. multiple or interacted absorbed FEs afterreghdfe), every Stataboottestcall must be followed byassert !missing(r(p))(check_inference.pyF10); whenboottestrefuses a multi-way or interacted FE specification, collapse each interacted FE pair into a single group id (egen fe_ab = group(a b)), verify the point estimate is identical, and re-run, or fall back to CV3/randomization. Always also run WCR-C and report its p beside WCR-S (MNW a