paper-review
Review academic papers for correctness, quality, and novelty using OpenJudge's multi-stage pipeline. Supports PDF files and LaTeX source packages (.tar.gz/.zip). Covers 10 disciplines: cs, medicine, physics, chemistry, biology, economics, psychology, environmental_science, mathematics, social_sciences. Use when the user asks to review, evaluate, critique, or assess a research paper, check references, or verify a BibTeX file.
agentscope-ai · 已有 AI 审核记录
安装与使用
运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。
npx skills add agentscope-ai/OpenJudge --skill paper-review技能说明
name: paper-review description: > Review academic papers for correctness, quality, and novelty using OpenJudge's multi-stage pipeline. Supports PDF files and LaTeX source packages (.tar.gz/.zip). Covers 10 disciplines: cs, medicine, physics, chemistry, biology, economics, psychology, environmental_science, mathematics, social_sciences. Use when the user asks to review, evaluate, critique, or assess a research paper, check references, or verify a BibTeX file.
Paper Review Skill
Multi-stage academic paper review using the OpenJudge PaperReviewPipeline:
- Safety check — jailbreak detection + format validation
- Correctness — objective errors (math, logic, data inconsistencies)
- Review — quality, novelty, significance (score 1–6)
- Criticality — severity of correctness issues
- BibTeX verification — cross-checks references against CrossRef/arXiv/DBLP
Prerequisites
# Install OpenJudge
pip install py-openjudge
# Extra dependency for paper_review
pip install litellm
pip install pypdfium2 # only if using vision mode (use_vision_for_pdf=True)
Gather from user before running
| Info | Required? | Notes |
|---|---|---|
| Paper file path | Yes | PDF or .tar.gz/.zip TeX package |
| API key | Yes | Env var preferred: OPENAI_API_KEY, ANTHROPIC_API_KEY, etc. |
| Model name | No | gpt-5.2, anthropic/claude-opus-4-6, dashscope/qwen-vl-plus. See Model selection below |
| Discipline | No | If not given, uses general CS/ML-oriented prompts |
| Venue | No | e.g. "NeurIPS 2025", "The Lancet" |
| Instructions | No | Free-form reviewer guidance, e.g. "Focus on experimental design" |
| Language | No | "en" (default) or "zh" for Simplified Chinese output |
| BibTeX file | No | Required only for reference verification |
| CrossRef email | No | Improves API rate limits for BibTeX verification |
Quick start
File type is auto-detected: .pdf → PDF review, .tar.gz/.zip → TeX review, .bib → BibTeX verification.
# Basic PDF review
python -m cookbooks.paper_review paper.pdf
# With discipline and venue
python -m cookbooks.paper_review paper.pdf \
--discipline cs --venue "NeurIPS 2025"
# Chinese output
python -m cookbooks.paper_review paper.pdf --language zh
# Custom reviewer instructions
python -m cookbooks.paper_review paper.pdf \
--instructions "Focus on experimental design and reproducibility"
# PDF + BibTeX verification
python -m cookbooks.paper_review paper.pdf \
--bib references.bib --email your@email.com
# Vision mode (for models that prefer images over text extraction)
python -m cookbooks.paper_review paper.pdf \
--vision --vision_max_pages 30 --format_vision_max_pages 10
# TeX source package
python -m cookbooks.paper_review paper_source.tar.gz \
--discipline biology --email your@email.com
# TeX source package with Chinese output and custom instructions
python -m cookbooks.paper_review paper_source.tar.gz \
--language zh --instructions "This is a short paper, be concise"
# Verify a standalone BibTeX file
python -m cookbooks.paper_review --bib_only references.bib --email your@email.com
All options
| Flag | Default | Description |
|---|---|---|
input (positional) | — | Path to PDF, TeX package, or .bib file |
--bib_only | — | Path to .bib file for standalone verification (no review) |
--model | gpt-4o | Model name |
--api_key | env var | API key |
--base_url | — | Custom API endpoint — must end at /v1, not /v1/chat/completions (litellm appends the path automatically) |
--discipline | — | Academic discipline |
--venue | — | Target conference/journal |
--instructions | — | Free-form reviewer guidance |
--language | en | Output language: en or zh |
--bib | — | Path to .bib file (for PDF review + reference verification) |
--email | — | CrossRef mailto for BibTeX check |
--paper_name | filename stem | Paper title in report |
--output | auto | Output .md report path |
--no_safety | off | Skip safety checks |
--no_correctness | off | Skip correctness check |
--no_criticality | off | Skip criticality verification |
--no_bib | off | Skip BibTeX verification |
--vision | on | Use vision mode (requires pypdfium2); enabled by default |
--vision_max_pages | 30 | Max pages in vision mode (0 = all) |
--format_vision_max_pages | 10 | Max pages for format check (0 = use --vision_max_pages) |
--timeout | 7500 | API timeout in seconds |
Interpreting results
Review score (1–6):
- 1–2: Reject (major flaws or well-known results)
- 3: Borderline reject
- 4: Borderline accept
- 5–6: Accept / Strong accept
Correctness score (1–3):
- 1: No objective errors
- 2: Minor errors (notation, arithmetic in non-critical parts)
- 3: Major errors (wrong proofs, core algorithm flaws)
BibTeX verification:
verified: found in CrossRef/arXiv/DBLPsuspect: title/author mismatch or not found — manual check recommended
Model selection
This pipeline uses litellm for model calls. Provider prefixes are handled automatically by the pipeline — see the table below.
IMPORTANT: The model MUST support multimodal (vision) input. PDF review uses vision mode
(--vision) to render pages as images, which requires a vision-capable model. Text-only models
will fail or produce empty reviews.
The --model value uses a provider/model-name convention so the pipeline knows
which API endpoint to call. The table below shows the exact string to pass:
| Provider | --model value | Env var | Notes |
|---|---|---|---|
| OpenAI | gpt-5.2, gpt-5-mini, … | OPENAI_API_KEY | No prefix needed; gpt-5.2 is the current flagship vision model; check OpenAI models for the latest |
| Anthropic | anthropic/claude-opus-4-6, `anthropic/claude-son |