Skip to content
OpenAgentSkill
Browsing is open · Catalog captured 2026-11-21 · Automatic updates and account services are paused.
Development

quantitative-model-diagnostics

Diagnostics for calibrated and quantitative-theory results — a headline number ("X explains 40% of Y") produced by solving or simulating a model. Inventories the free parameters, sweeps the calibration over defensible ranges, tests fit on non-targeted moments, and checks the code solves the model the text describes. Invoke from `code-and-data` whenever a load- bearing magnitude comes out of a calibrated/simulated model. The fragility sweep, applied to a model instead of a regression.

isitcredible · Static checks on file

Installation

Read the source and dependencies before running a command. Copying does not install or execute anything.

npx skills add isitcredible/research_skills --skill quantitative-model-diagnostics

About this skill


name: quantitative-model-diagnostics description: >- Diagnostics for calibrated and quantitative-theory results — a headline number ("X explains 40% of Y") produced by solving or simulating a model. Inventories the free parameters, sweeps the calibration over defensible ranges, tests fit on non-targeted moments, and checks the code solves the model the text describes. Invoke from code-and-data whenever a load- bearing magnitude comes out of a calibrated/simulated model. The fragility sweep, applied to a model instead of a regression.

Quantitative-model diagnostics

A calibrated model is a machine for turning parameter choices into a headline number. Some of those parameters are pinned to external evidence; some are set inside the model to hit chosen targets; some are simply assumed. The result's credibility is the credibility of those choices — and a number that holds only at a knife-edge value, or that its own model reproduces only because it was built to, is fragile in exactly the way a coefficient that lives in one influential region is fragile. This module puts that fragility on the record.

Trigger: any target result whose magnitude is produced by solving, simulating, or calibrating a model — a quantitative-theory exercise, a structural decomposition, a "our model accounts for X%" claim, the quantitative section of a hybrid theory paper. NOT_APPLICABLE when no load-bearing number comes out of a model; say so and name the estimates instead.

Boundaries. Where the model is proved as well as calibrated, this module runs alongside theory-diagnostics (it audits the numbers; that module audits the proof). Where structural parameters are estimated on data with standard errors (GMM/SMM/MLE), the empirical inference and identification checks still apply through the ordinary battery. Where the model is fitted to trending aggregates over time, time-series-diagnostics fires too — minimizing fit residuals does not guarantee stationarity; fitted-residual tests need reference distributions that account for estimation and selection, and in-sample fit alone is not evidence of fit.

Method crib (source-audited): ../../references/methods/andrews_gentzkow_shapiro_2017_moment_sensitivity.md — the Λ sensitivity matrix: which data moments each estimated parameter loads on, at negligible cost. A structural estimate whose headline loads on a few disputed moments is fragile in a way no standard error shows; a model that cannot report its moment loadings asks the reader to take identification on faith. This is the structural analogue of Rotemberg weights — name the load-bearing moments, then check them.

Discipline — a fragility claim is a re-run, never an assertion

Every number here traces to a script and a log in the job tree, re-run clean before it is reported (the battery's reproducibility rule is not waived for a model). You may not call a result knife-edge, or a moment mismatched, from reading the calibration table. Run the sweep, pull the range from the output, and report it. Equally: if the headline survives every defensible perturbation, report that it survived — a robust quantitative result is a strong one, and a manufactured failure is a failed check.

Core checks — run for every model-produced target

  • Free-parameter inventory, with anchor provenance. For every parameter, record: its value, its source (externally calibrated to a cited estimate / internally set to match a stated target / assumed with no external anchor), whether the headline result depends on it, and the status of its anchor. An external citation must measure the SAME quantity under the same definition — same horizon, population, units; a micro elasticity anchoring a macro parameter is the classic mismatch, and a mismatched anchor is a named finding, not a citation. An anchor the battery or the data audit has impeached demotes every parameter resting on it to "assumed with no anchor": bar "the model matches its targets" language until the internal block is re-solved at the target's defensible interval, and an impeached untargeted moment cannot count as overfit evidence either. The assumed-with-no-anchor parameters are where the result is most exposed; name them explicitly. A parameter "disciplined by the data" that is actually free to hit the target is an internal one mislabelled.
  • Calibration-sensitivity sweep. Perturb each load-bearing parameter across its defensible range — the spread of estimates in the literature, not only what the paper cites, or plausible bounds where none is cited — and report the headline quantity's range, one parameter at a time and then jointly across the parameters the result is most sensitive to. State the result in the paper's own units ("the 40% share ranges 18%–71% across defensible calibrations"). A number that collapses off a single point is the finding. State the sweep convention: externally-calibrated parameters are swept with the internal block re-matched to targets, or explicitly held fixed — say which, since holding the block fixed conflates parameter sensitivity with mechanical target-breaking. For internally-set parameters report target-uncertainty propagation: re-solve for the parameter at the target's interval endpoints. Report the breaking set, not just the range — the parameter values at which the claim fails and whether the defensible range intersects it ("breaks only outside the defensible range" vs "breaks inside it"). Count solver outcomes: N-solved/N-attempted per sweep arm — silently dropped no-solution cells manufacture robustness, no-solution cells counted as movement manufacture a kill, and under equilibrium multiplicity state which branch the sweep rides. Before a "robust" verdict, screen interactions over the parameters the headline depends on (a two-way grid over the top-k at minimum): a fragility living in a two-parameter interaction
Read the complete source ↗