synthetic-control-diagnostics
Diagnostics for synthetic control and synthetic difference-in-differences: pre-period adequacy, placebo-unit rank, weight diagnostics including lagged-outcome degeneracy, terminal-period sensitivity, and in-time placebos. Invoke from `code-and-data` whenever a target result uses a synthetic comparison unit.
isitcredible · 已有静态检查记录
安装与使用
运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。
npx skills add isitcredible/research_skills --skill synthetic-control-diagnostics技能说明
name: synthetic-control-diagnostics
description: >-
Diagnostics for synthetic control and synthetic difference-in-differences:
pre-period adequacy, placebo-unit rank, weight diagnostics including
lagged-outcome degeneracy, terminal-period sensitivity, and in-time
placebos. Invoke from code-and-data whenever a target result uses a
synthetic comparison unit.
Synthetic-control diagnostics
Ask how the fit was achieved, how the treated unit ranks against placebos, and whether the gap survives the choice of endpoint. A synthetic control is only as good as its donor weights, and the weights can be manufactured: match on enough lagged outcomes and the fit becomes mechanical while the covariates become decoration — so a tight pre-period fit is where the questions start, not where they end.
Trigger: a synthetic control or synthetic difference-in-differences
comparison behind any target result. NOT_APPLICABLE otherwise.
Method crib (source-audited):
../../references/methods/synthetic_control_guide.md — Abadie-Diamond-Hainmueller fit/placebo
machinery, Ferman spec-search accounting, Kaul all-lags pathology, BMF
augmentation, CWZ conformal, scpi prediction intervals, SDID.
../../references/methods/cattaneo_feng_titiunik_2021_sc_intervals.md —
JASA: conditional prediction intervals on the counterfactual itself.
The permutation rank test answers "is this gap unusual among placebos?",
not "what is the CI on the effect?". The CFT interval is around the
synthetic counterfactual: the test is whether it contains the treated
unit's observed outcome. A tight rank p with a treated outcome inside the
interval means the effect is not distinguishable from prediction error under
CFT's assumptions — a test under rule 8 (report its power), not a correction; a
wide interval that still excludes the treated outcome is not. Report the interval alongside the
rank.
Core checks
- Pre-period adequacy — and route the answer. Is the pre-treatment window long enough to discipline the fit and support the chosen inference procedure? A short pre-period plus many donors is a recipe for overfitting the fit and underpowering the test. When T0 is short relative to the donor count or the post-period, a tight pre-fit is manufactured evidence: require a penalized/augmented estimator and report the placebo grid's resolution limit; when T0 is adequate, unlock the in-time placebo and the pre-period holdout below. State the rule you applied.
- Placebo-unit distribution and rank. Run the full procedure on every
donor as if it were treated; report where the treated unit's effect ranks
in that distribution. Specify the statistic: the canonical ranking is the
post/pre RMSPE ratio (Abadie–Diamond–Hainmueller), not the raw gap —
donors with poor pre-fit produce large mechanical gaps and contaminate the
null. Screen the pool: state the treated unit's own pre-RMSPE and exclude
placebos whose pre-RMSPE exceeds a stated multiple of it. Then report J
and the smallest attainable p — the grid is 1/(J+1), and when that exceeds
α the test cannot reject and the record is "resolution-limited,
uninformative," not a null; symmetrically, a grid-minimum rank is not a
clean p-value. The rank is the design-based test — a treated effect
inside the screened placebo distribution is no evidence of an effect
however dramatic the gap plot looks; read it beside the conformal and
scpiintervals below. - Weight diagnostics. Report the donor weights: sparse, extreme, or degenerate weights mean the "synthetic" unit is really one or two donors, and the design inherits their idiosyncrasies. Then check those donors for their own shocks in the treatment period, run a donor leave-one-out refit (drop each positive-weight donor, report the gap range), and screen the placebo pool for donors that received similar interventions — a contaminated placebo pool shrinks the apparent rank.
- Terminal-period sensitivity. Recompute the effect dropping the final periods; divergence that lives at the end of the sample is fragile to it.
- Alternative-estimator refit. Re-fit the headline gap under augmented SC (Ben-Michael–Feller–Rothstein — the standard repair for imperfect pre-fit), SDID, and/or generalized SC; report whether the gap survives the estimator, not just the donor set.
- Conformal inference. Chernozhukov–Wüthrich–Zhu conformal intervals are
the second design-based inference route beside the placebo rank and
scpi; where the placebo grid is coarse, conformal is often the higher-resolution test. - Model-based prediction intervals (
scpi). The placebo rank is one inference route;scpi(Cattaneo–Feng–Palomba–Titiunik) is a second, independent one — it puts a prediction interval around the synthetic counterfactual that carries the estimator's own in-sample and out-of-sample uncertainty. Report it beside the placebo rank: a gap that ranks extreme among placebos while the treated outcome sits inside the counterfactual'sscpiinterval is not the clean effect the gap plot implies; a treated outcome outside a wide interval is still outside it. Re-run under alternative weight constraints (simplex, lasso, ridge, ols) and report whether the gap survives the constraint. (scpiimplements this — data prep,scest,scpi,scplot; see the battery's software examples.)
Conditional checks
- Lagged-outcome degeneracy — when lagged outcomes enter the matching variables: check whether they collapse the weights onto the outcome path, making the fit mechanical and the covariate matching irrelevant. Report the fit with and without the saturated lag set. This is a proved pathology, not a suspicion: Kaul, Klößner, Pfeifer & Schieler (2022, JBES) prove that when every pre-period outcome enters as a separate predictor, every other covariate is rendered irrelevant to the weights, however much it matters for the post-p