Skip to content
OpenAgentSkill
Browsing is open · Catalog captured 2026-11-21 · Automatic updates and account services are paused.
Development

review-paper-code

Review research code for reproducibility and quality, extract the paper's main empirical claims, compare paper to code, and write a constructive markdown report. Designed for social science / economics projects with LaTeX papers and Stata, R, or Python code.

claesbackman · AI review on file

Installation

Read the source and dependencies before running a command. Copying does not install or execute anything.

npx skills add claesbackman/AI-research-feedback --skill review-paper-code

About this skill


name: review-paper-code description: Review research code for reproducibility and quality, extract the paper's main empirical claims, compare paper to code, and write a constructive markdown report. Designed for social science / economics projects with LaTeX papers and Stata, R, or Python code. argument-hint: [optional: path/to/main.tex] [optional: path/to/code_dir] [optional: main|full] allowed-tools: Read, Write, Edit, Glob, Grep, Bash, Agent disable-model-invocation: true

Review Paper Code

Review a research project's paper and code for reproducibility, code quality, and paper-code alignment. Be constructive, concrete, and calibrated. Treat gaps as items to verify, not accusations.

Scope

This skill supports:

  • LaTeX papers
  • Stata (.do), R (.R, .r), and Python (.py) code

Default review depth:

  • main: prioritize the main paper, main scripts, and core outputs
  • full: inspect all detected code files in scope

If no depth is provided, default to main.

Phase 1: Discover the Project

First parse $ARGUMENTS:

  • If one argument looks like a .tex path, use it as PAPER_FILE.
  • If one argument looks like a directory path, use it as CODE_DIR.
  • If one argument is main or full, use it as REVIEW_DEPTH.

If any of the above are missing, auto-detect them.

1. Find the paper

Use Glob to search for **/*.tex, excluding obvious build folders such as _minted-*, build/, output/, .git/, node_modules/.

Identify the main paper file as the best candidate containing \documentclass or \begin{document}.

If multiple candidates exist, first discard files whose document class is beamer (slides) and files whose name or folder suggests an old draft or a response letter (response*, letter*, slides*, old*, archive/, etc.). Then prefer:

  1. A path explicitly provided in $ARGUMENTS
  2. A file in Writing/, writing/, Paper/, paper/, Draft/, or the repo root
  3. The file that appears to include the most component files via \input{} / \include{}

Record the result as PAPER_FILE.

2. Find the code

If CODE_DIR was not provided, look for likely code roots in this order:

  • Code/
  • Analysis/
  • code/
  • analysis/
  • scripts/
  • src/
  • programs/
  • replication/

If no single directory is clearly best, use the repo root and limit later discovery to likely code files.

Record the result as CODE_DIR.

3. Find code files

Within CODE_DIR and subdirectories, find:

  • **/*.do
  • **/*.R
  • **/*.r
  • **/*.py

Exclude obvious caches, environments, and generated folders where appropriate.

If REVIEW_DEPTH = main, prioritize:

  • Master scripts such as main.do, master.do, run_all.R, main.R, main.py, run.py
  • Files referenced by those scripts
  • Files that generate tables, figures, or final datasets
  • If no master script exists, select the most central files and cap the initial review set at a reasonable number

If REVIEW_DEPTH = full, include all detected code files.

Record:

  • CODE_FILES_ALL
  • CODE_FILES_REVIEWED
  • languages present

4. Find supporting documentation

Look for:

  • README.md, README.txt, readme.md
  • requirements.txt, environment.yml, pyproject.toml
  • renv.lock, DESCRIPTION

Record relevant files as available.

5. Handle ambiguity gracefully

If you find a paper and at least some code, continue even if discovery is imperfect.

Only stop if you cannot find either:

  • a main paper file, or
  • any relevant Stata, R, or Python code files

If you stop, tell the user briefly what was missing and what paths they can pass explicitly.

Before proceeding, tell the user:

  • the paper file chosen
  • the code directory chosen
  • the number of code files detected and the number selected for review
  • the review depth
  • any ambiguity worth noting

Phase 2: Read the Paper

Read PAPER_FILE.

Recursively read files referenced by:

  • \input{}
  • \include{}
  • \subfile{}

Extract a compact working summary for later cross-checking:

  • Paper title
  • Main research question
  • Main sample description
  • Main data sources
  • Main dependent variables
  • Main explanatory variables or treatments
  • Main estimation methods
  • Fixed effects and clustering, if stated
  • Main sample restrictions
  • Main tables and figures only
  • Headline quantitative claims only

Do not try to extract every statistic in the paper. Prioritize the main empirical design and the outputs most likely to map to code.

Store this as PAPER_SUMMARY.

Phase 3: Launch 2 Agents in Parallel

In a single message, launch both agents using the Agent tool with subagent_type: "general-purpose".

Each agent must produce a compact, high-signal output. Do not ask for exhaustive per-file prose on every file unless the project is very small.


AGENT A: Code Reproducibility and Quality

Store as CODE_REVIEW_SUMMARY.

Prompt:

You are reviewing research code for reproducibility and code quality in a social science / economics project.

Files in scope:

  • Reviewed code files: [insert CODE_FILES_REVIEWED]
  • README / documentation files: [insert discovered supporting files or "none found"]

Review ONLY the files in scope. Do not use Glob or Grep to discover other files, and ignore any previous review reports (code_review_report*.md, PRE_SUBMISSION_REVIEW_*.md, QUICK_REVIEW_*.md, anything in a reviews/ folder) — they must not influence your review.

Review the files and produce a compact report focused on the most decision-relevant findings.

Check:

  1. Hardcoded absolute paths or machine-specific assumptions
  2. Randomized procedures without an obvious seed in local or upstream execution context
  3. Outputs that appear to be consumed but not obviously generated in the reviewed pipeline
  4. Data inputs and whether path conventions are consistent
  5. Dependency management and software requirements
  6. Run order and presence of a master script or documented pipeline
  7. Large commented-out blocks, w
Read the complete source ↗