Skip to content
OpenAgentSkill
Browsing is open · Catalog captured 2026-11-21 · Automatic updates and account services are paused.
Development

code-and-data

Second review stage: read the code against the text (with the formula map), run the replication package and reproduce its anchors, audit how the load-bearing variables were built from their sources, and stress-test every result family with the fragility battery and its design-specific modules.

isitcredible · Static checks on file

Installation

Read the source and dependencies before running a command. Copying does not install or execute anything.

npx skills add isitcredible/research_skills --skill code-and-data

About this skill


name: code-and-data description: >- Second review stage: read the code against the text (with the formula map), run the replication package and reproduce its anchors, audit how the load-bearing variables were built from their sources, and stress-test every result family with the fragility battery and its design-specific modules.

Code and data

Find out whether the code does what the paper says, whether the data are what the paper says, and whether each result survives defensible alternatives. A result that survives every valid check is reported as strong — the battery is a test, not a kill-generator. Contributions may be estimates, theorems (../modules/theory-diagnostics/), calibrated or simulated model numbers (../modules/quantitative-model-diagnostics/), or hybrids; classify each result family. Any of them read through a model also has its premises tested (../modules/whimsical-assumptions-diagnostics/).

Ground rules:

  • No finding without a construction — an executed regression, a counterexample, a re-run sweep, or a code-text mismatch cited at file:line (its effect on the printed cell run before it counts against a claim, where the code runs); never a verbal argument.
  • The anchor is the authors' code, executed in work/pkg/, every repair logged — failed repairs too, under "Tried and failed" in the job's PROGRESS.md. Missing generated intermediates are rebuilt from raw inputs, not treated as blockers.
  • A blocked check needs proof — exact file, command, error, attempted repair.
  • Verification is not reanalysis. "The deposited code does not produce the printed cell" is a narrow, high-confidence failure; "a different defensible specification gives a different answer" is a weaker disagreement. Keep them apart (Clemens, ../references/methods/clemens_2017_failed_replications.md).

A. Code against text (before running anything)

  1. Extract the claimed specifications — outcome, regressors, controls, fixed effects, every filter and sample restriction, weights, the variance estimator. Take them from the displayed coefficient rows, not the table notes.
  2. Trace the code to the lines that filter data and estimate models, and record every mismatch at file:line: hidden drops, undocumented trims, different controls/FE/clustering, convergence workarounds (raised iteration limits, relaxed tolerances, try/skip).
  3. Formula map — analysis/code-and-data/formula-map.csv, one row per printed formula: numbered and lettered equations, appendix definitions, and every data-construction rule (compounding, normalisation, index or share construction, deflators, back-casting): printed_ref | implementing file:line | identical/differs/not-implemented/no-code | exact discrepancy. Enumerate the printed side from the text. A printed formula with no row has not been audited.
  4. Mirror sweep — for every differs, grep the deposit for other implementations of the same transform. A slip reported only where first found understates its reach.
  5. Feasibility matrix — analysis/code-and-data/feasibility-matrix.csv: exhibit | required inputs | input status. Required before any "cannot be validated" claim; one genuine input falsifies the blanket.
  6. Provenance — every input the code touches gets its type from review/deposit-inventory.json; an unknown file is probed before it supports any "simulated" or "restricted" conclusion.
  7. Prescribe each correction precisely: the native baseline must reproduce print first; one named edit; the full set of printed cells the object feeds. A discrepancy in a shared input is checked once per result family that uses it.

B. Run the package and reproduce

  • Inventory software and resources; map every result family (headline, mechanism, balance, placebo, robustness, appendix — not just the headline).
  • Reproduce anchors natively: estimate, N, SE/p, fixed effects, weights, clustering. Tolerance: exact match on N and degrees of freedom; relative difference < 1e-4 on estimates and SEs, or agreement to the printed rounding. Outside tolerance → NONREPRODUCED, a finding with the difference and the repair log. The tolerance is fixed: never widen it, or drop a cell from the comparison, to turn a mismatch into a match. The one exception is a stochastic number whose seed the deposit omits: its value is NONREPRODUCED only if it lies outside the range of at least 199 seeded runs (a 1% false-alarm rate when the printed value is itself one draw); with fewer runs, report its rank among them and make no call. The missing seed is recorded as a documentation gap.
  • Reconcile code with paper cell by cell: what variation identifies the cell; outcome, transform and units as the table claims; every claimed subsample rebuilt against all published columns (a match matrix, not a diagonal); subset labels checked with n_dropped/n_kept (a bounding box is not a region); the code implements the cited method, not a lookalike. Mismatch triage: rounding, then base categories, then which variance estimator reproduces the print (tables mix them by row), then which code block produced the table — transcribing printed values from page images. Do not write "reproduces" before this comparison exists.

C. Data audit

Reproduction shows you have the authors' file, not that the file is right. Errors and fabrication live upstream in construction. A false fraud charge is the most damaging error this audit can make; credulity is the other failure.

Always (the lite pass, every empirical paper):

  • Fix the target (anchored headline specification, load-bearing variables, version).
  • Map each load-bearing variable's chain: stated source → extraction → transcription → cleaning → imputation → transformation → merge → estimation field. Identify the regressor numerically against its source (a per-capita series h
Read the complete source ↗