code-and-data
Second review stage: read the code against the text (with the formula map), run the replication package and reproduce its anchors, audit how the load-bearing variables were built from their sources, and stress-test every result family with the fragility battery and its design-specific modules.
isitcredible · 已有静态检查记录
安装与使用
运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。
npx skills add isitcredible/research_skills --skill code-and-data技能说明
name: code-and-data description: >- Second review stage: read the code against the text (with the formula map), run the replication package and reproduce its anchors, audit how the load-bearing variables were built from their sources, and stress-test every result family with the fragility battery and its design-specific modules.
Code and data
Find out whether the code does what the paper says, whether the data are what the
paper says, and whether each result survives defensible alternatives. A result that
survives every valid check is reported as strong — the battery is a test, not a
kill-generator. Contributions may be estimates, theorems
(../modules/theory-diagnostics/), calibrated or simulated model numbers
(../modules/quantitative-model-diagnostics/), or hybrids; classify each result
family. Any of them read through a model also has its premises tested
(../modules/whimsical-assumptions-diagnostics/).
Ground rules:
- No finding without a construction — an executed regression, a counterexample, a
re-run sweep, or a code-text mismatch cited at
file:line(its effect on the printed cell run before it counts against a claim, where the code runs); never a verbal argument. - The anchor is the authors' code, executed in
work/pkg/, every repair logged — failed repairs too, under "Tried and failed" in the job'sPROGRESS.md. Missing generated intermediates are rebuilt from raw inputs, not treated as blockers. - A blocked check needs proof — exact file, command, error, attempted repair.
- Verification is not reanalysis. "The deposited code does not produce the printed
cell" is a narrow, high-confidence failure; "a different defensible specification
gives a different answer" is a weaker disagreement. Keep them apart (Clemens,
../references/methods/clemens_2017_failed_replications.md).
A. Code against text (before running anything)
- Extract the claimed specifications — outcome, regressors, controls, fixed effects, every filter and sample restriction, weights, the variance estimator. Take them from the displayed coefficient rows, not the table notes.
- Trace the code to the lines that filter data and estimate models, and record
every mismatch at
file:line: hidden drops, undocumented trims, different controls/FE/clustering, convergence workarounds (raised iteration limits, relaxed tolerances, try/skip). - Formula map —
analysis/code-and-data/formula-map.csv, one row per printed formula: numbered and lettered equations, appendix definitions, and every data-construction rule (compounding, normalisation, index or share construction, deflators, back-casting):printed_ref | implementing file:line | identical/differs/not-implemented/no-code | exact discrepancy. Enumerate the printed side from the text. A printed formula with no row has not been audited. - Mirror sweep — for every
differs, grep the deposit for other implementations of the same transform. A slip reported only where first found understates its reach. - Feasibility matrix —
analysis/code-and-data/feasibility-matrix.csv:exhibit | required inputs | input status. Required before any "cannot be validated" claim; one genuine input falsifies the blanket. - Provenance — every input the code touches gets its type from
review/deposit-inventory.json; anunknownfile is probed before it supports any "simulated" or "restricted" conclusion. - Prescribe each correction precisely: the native baseline must reproduce print first; one named edit; the full set of printed cells the object feeds. A discrepancy in a shared input is checked once per result family that uses it.
B. Run the package and reproduce
- Inventory software and resources; map every result family (headline, mechanism, balance, placebo, robustness, appendix — not just the headline).
- Reproduce anchors natively: estimate, N, SE/p, fixed effects, weights,
clustering. Tolerance: exact match on N and degrees of freedom; relative difference
< 1e-4 on estimates and SEs, or agreement to the printed rounding. Outside tolerance
→
NONREPRODUCED, a finding with the difference and the repair log. The tolerance is fixed: never widen it, or drop a cell from the comparison, to turn a mismatch into a match. The one exception is a stochastic number whose seed the deposit omits: its value isNONREPRODUCEDonly if it lies outside the range of at least 199 seeded runs (a 1% false-alarm rate when the printed value is itself one draw); with fewer runs, report its rank among them and make no call. The missing seed is recorded as a documentation gap. - Reconcile code with paper cell by cell: what variation identifies the cell;
outcome, transform and units as the table claims; every claimed subsample rebuilt
against all published columns (a match matrix, not a diagonal); subset labels
checked with
n_dropped/n_kept(a bounding box is not a region); the code implements the cited method, not a lookalike. Mismatch triage: rounding, then base categories, then which variance estimator reproduces the print (tables mix them by row), then which code block produced the table — transcribing printed values from page images. Do not write "reproduces" before this comparison exists.
C. Data audit
Reproduction shows you have the authors' file, not that the file is right. Errors and fabrication live upstream in construction. A false fraud charge is the most damaging error this audit can make; credulity is the other failure.
Always (the lite pass, every empirical paper):
- Fix the target (anchored headline specification, load-bearing variables, version).
- Map each load-bearing variable's chain:
stated source → extraction → transcription → cleaning → imputation → transformation → merge → estimation field. Identify the regressor numerically against its source (a per-capita series h