review-paper-code
Review research code for reproducibility and quality, extract the paper's main empirical claims, compare paper to code, and write a constructive markdown report. Designed for social science / economics projects with LaTeX papers and Stata, R, or Python code.
claesbackman · AI review on file
Installation
Read the source and dependencies before running a command. Copying does not install or execute anything.
npx skills add claesbackman/AI-research-feedback --skill review-paper-codeAbout this skill
name: review-paper-code description: Review research code for reproducibility and quality, extract the paper's main empirical claims, compare paper to code, and write a constructive markdown report. Designed for social science / economics projects with LaTeX papers and Stata, R, or Python code. argument-hint: [optional: path/to/main.tex] [optional: path/to/code_dir] [optional: main|full] allowed-tools: Read, Write, Edit, Glob, Grep, Bash, Agent disable-model-invocation: true
Review Paper Code
Review a research project's paper and code for reproducibility, code quality, and paper-code alignment. Be constructive, concrete, and calibrated. Treat gaps as items to verify, not accusations.
Scope
This skill supports:
- LaTeX papers
- Stata (
.do), R (.R,.r), and Python (.py) code
Default review depth:
main: prioritize the main paper, main scripts, and core outputsfull: inspect all detected code files in scope
If no depth is provided, default to main.
Phase 1: Discover the Project
First parse $ARGUMENTS:
- If one argument looks like a
.texpath, use it asPAPER_FILE. - If one argument looks like a directory path, use it as
CODE_DIR. - If one argument is
mainorfull, use it asREVIEW_DEPTH.
If any of the above are missing, auto-detect them.
1. Find the paper
Use Glob to search for **/*.tex, excluding obvious build folders such as _minted-*, build/, output/, .git/, node_modules/.
Identify the main paper file as the best candidate containing \documentclass or \begin{document}.
If multiple candidates exist, first discard files whose document class is beamer (slides) and files whose name or folder suggests an old draft or a response letter (response*, letter*, slides*, old*, archive/, etc.). Then prefer:
- A path explicitly provided in
$ARGUMENTS - A file in
Writing/,writing/,Paper/,paper/,Draft/, or the repo root - The file that appears to include the most component files via
\input{}/\include{}
Record the result as PAPER_FILE.
2. Find the code
If CODE_DIR was not provided, look for likely code roots in this order:
Code/Analysis/code/analysis/scripts/src/programs/replication/
If no single directory is clearly best, use the repo root and limit later discovery to likely code files.
Record the result as CODE_DIR.
3. Find code files
Within CODE_DIR and subdirectories, find:
**/*.do**/*.R**/*.r**/*.py
Exclude obvious caches, environments, and generated folders where appropriate.
If REVIEW_DEPTH = main, prioritize:
- Master scripts such as
main.do,master.do,run_all.R,main.R,main.py,run.py - Files referenced by those scripts
- Files that generate tables, figures, or final datasets
- If no master script exists, select the most central files and cap the initial review set at a reasonable number
If REVIEW_DEPTH = full, include all detected code files.
Record:
CODE_FILES_ALLCODE_FILES_REVIEWED- languages present
4. Find supporting documentation
Look for:
README.md,README.txt,readme.mdrequirements.txt,environment.yml,pyproject.tomlrenv.lock,DESCRIPTION
Record relevant files as available.
5. Handle ambiguity gracefully
If you find a paper and at least some code, continue even if discovery is imperfect.
Only stop if you cannot find either:
- a main paper file, or
- any relevant Stata, R, or Python code files
If you stop, tell the user briefly what was missing and what paths they can pass explicitly.
Before proceeding, tell the user:
- the paper file chosen
- the code directory chosen
- the number of code files detected and the number selected for review
- the review depth
- any ambiguity worth noting
Phase 2: Read the Paper
Read PAPER_FILE.
Recursively read files referenced by:
\input{}\include{}\subfile{}
Extract a compact working summary for later cross-checking:
- Paper title
- Main research question
- Main sample description
- Main data sources
- Main dependent variables
- Main explanatory variables or treatments
- Main estimation methods
- Fixed effects and clustering, if stated
- Main sample restrictions
- Main tables and figures only
- Headline quantitative claims only
Do not try to extract every statistic in the paper. Prioritize the main empirical design and the outputs most likely to map to code.
Store this as PAPER_SUMMARY.
Phase 3: Launch 2 Agents in Parallel
In a single message, launch both agents using the Agent tool with subagent_type: "general-purpose".
Each agent must produce a compact, high-signal output. Do not ask for exhaustive per-file prose on every file unless the project is very small.
AGENT A: Code Reproducibility and Quality
Store as CODE_REVIEW_SUMMARY.
Prompt:
Read the complete source ↗You are reviewing research code for reproducibility and code quality in a social science / economics project.
Files in scope:
- Reviewed code files: [insert
CODE_FILES_REVIEWED]- README / documentation files: [insert discovered supporting files or "none found"]
Review ONLY the files in scope. Do not use Glob or Grep to discover other files, and ignore any previous review reports (
code_review_report*.md,PRE_SUBMISSION_REVIEW_*.md,QUICK_REVIEW_*.md, anything in areviews/folder) — they must not influence your review.Review the files and produce a compact report focused on the most decision-relevant findings.
Check:
- Hardcoded absolute paths or machine-specific assumptions
- Randomized procedures without an obvious seed in local or upstream execution context
- Outputs that appear to be consumed but not obviously generated in the reviewed pipeline
- Data inputs and whether path conventions are consistent
- Dependency management and software requirements
- Run order and presence of a master script or documented pipeline
- Large commented-out blocks, w