文档处理
pdf-parser
Convert PDF papers to markdown using PaddleOCR API
PKUfudawei · 已有 AI 审核记录
安装与使用
运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。
npx skills add PKUfudawei/arxiv-skills --skill pdf-parser技能说明
name: pdf-parser description: Convert PDF papers to markdown using PaddleOCR API license: MIT
pdf-parser
Convert PDF papers to markdown format using PaddleOCR-VL-1.5 API, with automatic image extraction and layout preservation.
Overview
This skill uses PaddleOCR's cloud API to convert PDF academic papers into markdown format. It preserves document structure, extracts text content, and downloads embedded images.
Workflow
PDF Files → PaddleOCR API → JSONL Result → Markdown + Images
Processing Steps
- Submit Job: Upload each PDF to PaddleOCR API
- Wait for Result: Poll job status until completion
- Download & Merge: Fetch JSONL result, extract markdown text and images
- Save Output: Write combined markdown file with embedded image references
Setup
1. Obtain PaddleOCR API Token
Get your API token from PaddleOCR AI Studio and set the environment variable:
export PADDLE_TOKEN="your_token_here"
Or add to ~/.bashrc or .env file for persistent configuration:
echo 'export PADDLE_TOKEN="your_token_here"' >> ~/.bashrc
source ~/.bashrc
Usage
Basic Usage
python3 skills/pdf-parser/scripts/paddleOCR.py -f "path/to/pdfs/*.pdf"
Process Multiple Directories
# Process all PDFs in arxiv subdirectories
python3 skills/pdf-parser/scripts/paddleOCR.py -f "arxiv/*/*.pdf"
# Process specific directory
python3 skills/pdf-parser/scripts/paddleOCR.py -f "/home/user/papers/*.pdf"
Args
| Argument | Short | Default | Description |
|---|---|---|---|
--files | -f | (required) | Glob pattern(s) for PDF files to process (e.g. "arxiv/*/*.pdf") |
--workers | 5 | Maximum parallel PDF processing threads |
Output Format
File Naming
- Input:
paper_123.pdf - Output:
paper_123.md(same directory as input)
Markdown Structure
- Text content extracted and preserved in reading order
- Images saved as separate files with references in markdown
- Layout structure maintained (headers, paragraphs, lists)
Image Handling
- Images extracted from PDF are downloaded separately
- Saved in the same directory as the markdown file
- Referenced in markdown using standard image syntax
Configuration
API Settings (Hardcoded)
| Setting | Value | Description |
|---|---|---|
MODEL | PaddleOCR-VL-1.5 | OCR model version |
useDocOrientationClassify | False | Disable auto-orientation |
useDocUnwarping | False | Disable document unwarping |
useChartRecognition | False | Disable chart recognition |
Concurrency Settings
| Parameter | Value | Description |
|---|---|---|
--workers | 5 | Maximum parallel PDF processing threads |
semaphore | 5 | Maximum concurrent API requests |
Example
Convert Single Directory
# Convert all PDFs in the papers directory
python3 skills/pdf-parser/scripts/paddleOCR.py -f "./papers/*.pdf"
# Output:
# ./papers/paper1.pdf → ./papers/paper1.md
# ./papers/paper2.pdf → ./papers/paper2.md
Convert arXiv Download Directory
# Typical workflow after arxiv-retriever download
python3 skills/pdf-parser/scripts/paddleOCR.py -f "arxiv/*/*.pdf"
# Output:
# arxiv/cs.LG/paper_123.pdf → arxiv/cs.LG/paper_123.md
Error Handling
- Skipped Files: PDFs that already have a corresponding
.mdfile are skipped - Failed Jobs: Server-side job failures / timeouts are reported with the job id via
tqdm.write(); all failed paths are collected in./failed_files.txtso the batch can be re-run - Network Errors: Request failures are caught and logged per-file
- Exit Code: the script exits non-zero when any file failed or errored
Performance
- Parallel Processing: Up to 5 PDFs processed concurrently by default (
--workers) - Rate Limiting: Max 5 simultaneous API requests to avoid throttling
- Progress Tracking: Real-time progress bar using
tqdm
Dependencies
requests- HTTP API callstqdm- Progress barspython-dotenv- Environment variable loading
Notes
- Requires internet connection for API calls
- API usage may be subject to quotas or billing
- Large PDFs may take longer to process
- Image files are downloaded separately and saved alongside markdown