跳至正文
OpenAgentSkill
浏览已开放 · 目录记录日期 2026-11-21 · 自动更新和账户服务暂时关闭。
文档处理

pdf-parser

Convert PDF papers to markdown using PaddleOCR API

PKUfudawei · 已有 AI 审核记录

安装与使用

运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。

npx skills add PKUfudawei/arxiv-skills --skill pdf-parser

技能说明


name: pdf-parser description: Convert PDF papers to markdown using PaddleOCR API license: MIT

pdf-parser

Convert PDF papers to markdown format using PaddleOCR-VL-1.5 API, with automatic image extraction and layout preservation.

Overview

This skill uses PaddleOCR's cloud API to convert PDF academic papers into markdown format. It preserves document structure, extracts text content, and downloads embedded images.

Workflow

PDF Files → PaddleOCR API → JSONL Result → Markdown + Images

Processing Steps

  1. Submit Job: Upload each PDF to PaddleOCR API
  2. Wait for Result: Poll job status until completion
  3. Download & Merge: Fetch JSONL result, extract markdown text and images
  4. Save Output: Write combined markdown file with embedded image references

Setup

1. Obtain PaddleOCR API Token

Get your API token from PaddleOCR AI Studio and set the environment variable:

export PADDLE_TOKEN="your_token_here"

Or add to ~/.bashrc or .env file for persistent configuration:

echo 'export PADDLE_TOKEN="your_token_here"' >> ~/.bashrc
source ~/.bashrc

Usage

Basic Usage

python3 skills/pdf-parser/scripts/paddleOCR.py -f "path/to/pdfs/*.pdf"

Process Multiple Directories

# Process all PDFs in arxiv subdirectories
python3 skills/pdf-parser/scripts/paddleOCR.py -f "arxiv/*/*.pdf"

# Process specific directory
python3 skills/pdf-parser/scripts/paddleOCR.py -f "/home/user/papers/*.pdf"

Args

ArgumentShortDefaultDescription
--files-f(required)Glob pattern(s) for PDF files to process (e.g. "arxiv/*/*.pdf")
--workers5Maximum parallel PDF processing threads

Output Format

File Naming

  • Input: paper_123.pdf
  • Output: paper_123.md (same directory as input)

Markdown Structure

  • Text content extracted and preserved in reading order
  • Images saved as separate files with references in markdown
  • Layout structure maintained (headers, paragraphs, lists)

Image Handling

  • Images extracted from PDF are downloaded separately
  • Saved in the same directory as the markdown file
  • Referenced in markdown using standard image syntax

Configuration

API Settings (Hardcoded)

SettingValueDescription
MODELPaddleOCR-VL-1.5OCR model version
useDocOrientationClassifyFalseDisable auto-orientation
useDocUnwarpingFalseDisable document unwarping
useChartRecognitionFalseDisable chart recognition

Concurrency Settings

ParameterValueDescription
--workers5Maximum parallel PDF processing threads
semaphore5Maximum concurrent API requests

Example

Convert Single Directory

# Convert all PDFs in the papers directory
python3 skills/pdf-parser/scripts/paddleOCR.py -f "./papers/*.pdf"

# Output:
# ./papers/paper1.pdf → ./papers/paper1.md
# ./papers/paper2.pdf → ./papers/paper2.md

Convert arXiv Download Directory

# Typical workflow after arxiv-retriever download
python3 skills/pdf-parser/scripts/paddleOCR.py -f "arxiv/*/*.pdf"

# Output:
# arxiv/cs.LG/paper_123.pdf → arxiv/cs.LG/paper_123.md

Error Handling

  • Skipped Files: PDFs that already have a corresponding .md file are skipped
  • Failed Jobs: Server-side job failures / timeouts are reported with the job id via tqdm.write(); all failed paths are collected in ./failed_files.txt so the batch can be re-run
  • Network Errors: Request failures are caught and logged per-file
  • Exit Code: the script exits non-zero when any file failed or errored

Performance

  • Parallel Processing: Up to 5 PDFs processed concurrently by default (--workers)
  • Rate Limiting: Max 5 simultaneous API requests to avoid throttling
  • Progress Tracking: Real-time progress bar using tqdm

Dependencies

  • requests - HTTP API calls
  • tqdm - Progress bars
  • python-dotenv - Environment variable loading

Notes

  • Requires internet connection for API calls
  • API usage may be subject to quotas or billing
  • Large PDFs may take longer to process
  • Image files are downloaded separately and saved alongside markdown