quantitative-model-diagnostics
Diagnostics for calibrated and quantitative-theory results — a headline number ("X explains 40% of Y") produced by solving or simulating a model. Inventories the free parameters, sweeps the calibration over defensible ranges, tests fit on non-targeted moments, and checks the code solves the model the text describes. Invoke from `code-and-data` whenever a load- bearing magnitude comes out of a calibrated/simulated model. The fragility sweep, applied to a model instead of a regression.
isitcredible · Static checks on file
Installation
Read the source and dependencies before running a command. Copying does not install or execute anything.
npx skills add isitcredible/research_skills --skill quantitative-model-diagnosticsAbout this skill
name: quantitative-model-diagnostics
description: >-
Diagnostics for calibrated and quantitative-theory results — a headline
number ("X explains 40% of Y") produced by solving or simulating a model.
Inventories the free parameters, sweeps the calibration over defensible
ranges, tests fit on non-targeted moments, and checks the code solves the
model the text describes. Invoke from code-and-data whenever a load-
bearing magnitude comes out of a calibrated/simulated model. The fragility
sweep, applied to a model instead of a regression.
Quantitative-model diagnostics
A calibrated model is a machine for turning parameter choices into a headline number. Some of those parameters are pinned to external evidence; some are set inside the model to hit chosen targets; some are simply assumed. The result's credibility is the credibility of those choices — and a number that holds only at a knife-edge value, or that its own model reproduces only because it was built to, is fragile in exactly the way a coefficient that lives in one influential region is fragile. This module puts that fragility on the record.
Trigger: any target result whose magnitude is produced by solving, simulating,
or calibrating a model — a quantitative-theory exercise, a structural
decomposition, a "our model accounts for X%" claim, the quantitative section of
a hybrid theory paper. NOT_APPLICABLE when no load-bearing number comes out of
a model; say so and name the estimates instead.
Boundaries. Where the model is proved as well as calibrated, this module runs
alongside theory-diagnostics (it audits the numbers; that module audits
the proof). Where structural parameters are estimated on data with standard
errors (GMM/SMM/MLE), the empirical inference and identification checks still
apply through the ordinary battery. Where the model is fitted to trending
aggregates over time, time-series-diagnostics fires too — minimizing fit residuals does not guarantee stationarity; fitted-residual tests need reference distributions that account for estimation and selection, and in-sample fit alone is not evidence
of fit.
Method crib (source-audited):
../../references/methods/andrews_gentzkow_shapiro_2017_moment_sensitivity.md —
the Λ sensitivity matrix: which data moments each estimated parameter
loads on, at negligible cost. A structural estimate whose headline loads
on a few disputed moments is fragile in a way no standard error shows;
a model that cannot report its moment loadings asks the reader to take
identification on faith. This is the structural analogue of Rotemberg
weights — name the load-bearing moments, then check them.
Discipline — a fragility claim is a re-run, never an assertion
Every number here traces to a script and a log in the job tree, re-run clean before it is reported (the battery's reproducibility rule is not waived for a model). You may not call a result knife-edge, or a moment mismatched, from reading the calibration table. Run the sweep, pull the range from the output, and report it. Equally: if the headline survives every defensible perturbation, report that it survived — a robust quantitative result is a strong one, and a manufactured failure is a failed check.
Core checks — run for every model-produced target
- Free-parameter inventory, with anchor provenance. For every parameter, record: its value, its source (externally calibrated to a cited estimate / internally set to match a stated target / assumed with no external anchor), whether the headline result depends on it, and the status of its anchor. An external citation must measure the SAME quantity under the same definition — same horizon, population, units; a micro elasticity anchoring a macro parameter is the classic mismatch, and a mismatched anchor is a named finding, not a citation. An anchor the battery or the data audit has impeached demotes every parameter resting on it to "assumed with no anchor": bar "the model matches its targets" language until the internal block is re-solved at the target's defensible interval, and an impeached untargeted moment cannot count as overfit evidence either. The assumed-with-no-anchor parameters are where the result is most exposed; name them explicitly. A parameter "disciplined by the data" that is actually free to hit the target is an internal one mislabelled.
- Calibration-sensitivity sweep. Perturb each load-bearing parameter across its defensible range — the spread of estimates in the literature, not only what the paper cites, or plausible bounds where none is cited — and report the headline quantity's range, one parameter at a time and then jointly across the parameters the result is most sensitive to. State the result in the paper's own units ("the 40% share ranges 18%–71% across defensible calibrations"). A number that collapses off a single point is the finding. State the sweep convention: externally-calibrated parameters are swept with the internal block re-matched to targets, or explicitly held fixed — say which, since holding the block fixed conflates parameter sensitivity with mechanical target-breaking. For internally-set parameters report target-uncertainty propagation: re-solve for the parameter at the target's interval endpoints. Report the breaking set, not just the range — the parameter values at which the claim fails and whether the defensible range intersects it ("breaks only outside the defensible range" vs "breaks inside it"). Count solver outcomes: N-solved/N-attempted per sweep arm — silently dropped no-solution cells manufacture robustness, no-solution cells counted as movement manufacture a kill, and under equilibrium multiplicity state which branch the sweep rides. Before a "robust" verdict, screen interactions over the parameters the headline depends on (a two-way grid over the top-k at minimum): a fragility living in a two-parameter interaction