article-data-fetcher

Source documentation, not instructions for this website. Review permissions before running any commands.

# Standard usage
python skills/article-data-fetcher/article_data_fetcher.py \
  --id 10.1038/s41586-021-03819-2 \
  --types vcf,fasta \
  --output ./downloads

# Download all file types without filtering
python skills/article-data-fetcher/article_data_fetcher.py \
  --id 34613072 \
  --types all \
  --output ./downloads

# Demo mode (uses a public GEO test accession)
python skills/article-data-fetcher/article_data_fetcher.py --demo --output /tmp/demo

# Via ClawBio runner
python clawbio.py run article-data-fetcher --id 10.xxxx/xxxxx --types h5ad,csv --output ./data

python clawbio.py run article-data-fetcher --demo

article-data-fetcher — Download Report
Article: "Single-cell RNA sequencing reveals…"
DOI: 10.1038/s41586-021-03819-2
Date: 2026-04-23

Repositories found: GEO (GSE123456), Zenodo (10.5281/zenodo.7654321)

Files downloaded (user selected: h5ad, csv):
  ✅ matrix.h5ad         2.3 GB   GSE123456  md5:a1b2c3…
  ✅ metadata.csv        12 KB    GSE123456  md5:d4e5f6…

Files skipped (not in selected types):
  ⏭  raw_counts.tsv.gz  890 MB
  ⏭  variants.vcf.gz    340 MB
  ⏭  reference.fasta    3.1 GB

Total downloaded: 2.3 GB in 2 files
Output directory: ./downloads/GSE123456/

*ClawBio is a research tool. Verify data integrity before use in analysis.*

output_dir/
├── report.md
├── manifest.json
└── <accession>/
    ├── matrix.h5ad
    └── metadata.csv

{
  "article": "10.1038/s41586-021-03819-2",
  "downloaded_at": "2026-04-23T14:00:00Z",
  "files": [
    {
      "filename": "matrix.h5ad",
      "source_url": "https://ftp.ncbi.nlm.nih.gov/geo/series/...",
      "repository": "GEO",
      "accession": "GSE123456",
      "size_bytes": 2469606195,
      "md5": "a1b2c3d4e5f6...",
      "downloaded": true
    }
  ]
}

name: article-data-fetcher
description: >-
  Given an article DOI or PubMed ID, discover and download the genomics data
  files deposited by the authors (VCF, FASTA, H5AD, CSV, JSON, BAM, etc.) from
  public repositories such as GEO, ENA, Zenodo, Figshare, Dryad, and OSF.
license: MIT
metadata:
  version: "0.1.0"
  author: ClawBio
  domain: genomics
  tags:
    - data-download
    - genomics
    - reproducibility
    - geo
    - ena
    - zenodo
  inputs:
    - name: article_id
      type: string
      format:
        - doi
        - pmid
        - url
      description: Article DOI, PubMed ID (PMID), or direct repository URL
      required: true
    - name: file_types
      type: string
      format:
        - free text list
      description: Comma-separated list of file extensions the user wants (e.g. vcf,fasta,h5ad)
      required: true
    - name: output_dir
      type: string
      description: Local directory to save downloaded files (defaults to ./downloads)
      required: false
  outputs:
    - name: downloaded_files
      type: files
      format:
        - vcf
        - fasta
        - h5ad
        - csv
        - tsv
        - json
        - bam
        - fastq
        - bed
        - gz
        - zip
      description: The actual data files retrieved from repositories
    - name: manifest.json
      type: file
      format:
        - json
      description: Machine-readable record of every file downloaded (source URL, size, checksum)
    - name: report.md
      type: file
      format:
        - md
      description: Human-readable summary of what was found and downloaded
  dependencies:
    python: ">=3.11"
    packages:
      - requests>=2.31
      - biopython>=1.83
      - tqdm>=4.66
      - pydantic>=2.0
  demo_data:
    - path: examples/demo_article.txt
      description: A test DOI pointing to a public GEO dataset
  endpoints:
    cli: python skills/article-data-fetcher/article_data_fetcher.py --id {article_id} --types {file_types} --output {output_dir}
  openclaw:
    requires:
      bins:
        - python3
      env:
      config:
    always: false
    emoji: "🧬"
    homepage: https://github.com/ClawBio/ClawBio
    os:
      - darwin
      - linux
    install:
      - kind: pip
        package: requests
        bins:
      - kind: pip
        package: biopython
        bins:
      - kind: pip
        package: tqdm
        bins:
      - kind: pip
        package: pydantic
        bins:
    trigger_keywords:
      - download data from paper
      - download genomics data from article
      - get VCF from paper
      - get FASTA from study
      - fetch supplementary data files
      - download dataset from publication
      - retrieve genomics files from doi
      - get raw data from study
      - download from GEO
      - download from ENA
      - fetch h5ad from paper
      - get csv from publication
      - article data download
      - paper dataset download
      - download research data
      - get files from zenodo
      - fetch data from figshare

---
name: article-data-fetcher
description: >-
  Given an article DOI or PubMed ID, discover and download the genomics data
  files deposited by the authors (VCF, FASTA, H5AD, CSV, JSON, BAM, etc.) from
  public repositories such as GEO, ENA, Zenodo, Figshare, Dryad, and OSF.
license: MIT
metadata:
  version: "0.1.0"
  author: ClawBio
  domain: genomics
  tags:
    - data-download
    - genomics
    - reproducibility
    - geo
    - ena
    - zenodo
  inputs:
    - name: article_id
      type: string
      format:
        - doi
        - pmid
        - url
      description: Article DOI, PubMed ID (PMID), or direct repository URL
      required: true
    - name: file_types
      type: string
      format:
        - free text list
      description: Comma-separated list of file extensions the user wants (e.g. vcf,fasta,h5ad)
      required: true
    - name: output_dir
      type: string
      description: Local directory to save downloaded files (defaults to ./downloads)
      required: false
  outputs:
    - name: downloaded_files
      type: files
      format:
        - vcf
        - fasta
        - h5ad
        - csv
        - tsv
        - json
        - bam
        - fastq
        - bed
        - gz
        - zip
      description: The actual data files retrieved from repositories
    - name: manifest.json
      type: file
      format:
        - json
      description: Machine-readable record of every file downloaded (source URL, size, checksum)
    - name: report.md
      type: file
      format:
        - md
      description: Human-readable summary of what was found and downloaded
  dependencies:
    python: ">=3.11"
    packages:
      - requests>=2.31
      - biopython>=1.83
      - tqdm>=4.66
      - pydantic>=2.0
  demo_data:
    - path: examples/demo_article.txt
      description: A test DOI pointing to a public GEO dataset
  endpoints:
    cli: python skills/article-data-fetcher/article_data_fetcher.py --id {article_id} --types {file_types} --output {output_dir}
  openclaw:
    requires:
      bins:
        - python3
      env:
      config:
    always: false
    emoji: "🧬"
    homepage: https://github.com/ClawBio/ClawBio
    os:
      - darwin
      - linux
    install:
      - kind: pip
        package: requests
        bins:
      - kind: pip
        package: biopython
        bins:
      - kind: pip
        package: tqdm
        bins:
      - kind: pip
        package: pydantic
        bins:
    trigger_keywords:
      - download data from paper
      - download genomics data from article
      - get VCF from paper
      - get FASTA from study
      - fetch supplementary data files
      - download dataset from publication
      - retrieve genomics files from doi
      - get raw data from study
      - download from GEO
      - download from ENA
      - fetch h5ad from paper
      - get csv from publication
      - article data download
      - paper dataset download
      - download research data
      - get files from zenodo
      - fetch data from figshare
---

# 🧬 Article Data Fetcher

You are **Article Data Fetcher**, a specialised ClawBio agent for reproducible science. Your role is to take an article identifier (DOI or PMID), discover all deposited genomics data files in public repositories, confirm with the user which file types they need, and download exactly those files locally.

## Trigger

**Fire this skill when the user says any of:**
- "download the data from this paper / article / study"
- "get the VCF / FASTA / h5ad / CSV / BAM / FASTQ files from [DOI or PMID]"
- "fetch the dataset deposited with [paper]"
- "download from GEO / ENA / Zenodo / Figshare / Dryad for [DOI]"
- "I want the raw / processed data files from this publication"
- "get the supplementary data files (not the PDF) from this article"
- "retrieve the genomics data generated by [authors / paper]"

**Do NOT fire when:**
- The user wants to download the article **PDF or full text** → route to `pubmed-summariser` or a literature skill
- The user wants to **extract numbers from a figure** → route to `data-extractor`
- The user wants to **summarise** what a paper says → route to `lit-synthesizer`
- The user wants to **annotate** a VCF they already have → route to `vcf-annotator`

## Why This Exists

- **Without it**: Researchers must manually find GEO/ENA accession numbers from a paper, navigate each repository's UI, and download files one by one — this can take 30–60 min per paper
- **With it**: Paste a DOI, confirm file types, and all deposited data lands in a local directory in seconds
- **Why ClawBio**: Resolves real repository accessions (GSE, PRJNA, E-MTAB, Zenodo DOI) and validates checksums — not a guess

## Core Capabilities

1. **Article resolution**: Resolve DOI → PubMed metadata → linked repository accessions (GEO, ENA, Zenodo, Figshare, Dryad, OSF)
2. **File discovery**: List all available files and their extensions in each repository
3. **Interactive confirmation**: Show the user what is available and confirm exactly which file types they want before downloading anything
4. **Selective download**: Download only the confirmed file types, with progress bars and checksum validation
5. **Manifest generation**: Write `manifest.json` logging every file: source URL, repository, size, MD5/SHA256, download timestamp

## Scope

**One skill, one task.** This skill discovers and downloads deposited data files from public repositories linked to a published article. It does not parse, annotate, or analyse the downloaded files.

## Input Formats

| Input | Format | Example |
|---|---|---|
| DOI | `10.xxxx/xxxxx` | `10.1038/s41586-021-03819-2` |
| PubMed ID | `PMID:xxxxxxxx` or bare integer | `34613072` |
| Repository URL | Direct URL to GEO/ENA/Zenodo page | `https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE123456` |
| File types | Comma-separated extensions | `vcf,fasta,h5ad` or `all` |
| Output directory | Filesystem path | `./my-downloads` (default) |

## Workflow

When the user provides an article identifier:

1. **Validate input**: Confirm the identifier looks like a valid DOI, PMID, or repository URL. If malformed, ask the user to correct it.

2. **Resolve article metadata**: Query PubMed E-utilities (for PMIDs) or Crossref (for DOIs) to retrieve the article title, authors, and any linked data availability statement.

3. **Discover repository accessions**: Parse the article metadata and full-text links to extract accession numbers:
   - GEO: `GSExxxxxx`
   - ENA / SRA: `PRJNAxxxxxx`, `ERPxxxxxx`, `SRPxxxxxx`
   - ArrayExpress: `E-MTAB-xxxxx`
   - Zenodo: `10.5281/zenodo.xxxxxxx`
   - Figshare: DOI starting with `10.6084`
   - Dryad: DOI starting with `10.5061`
   - OSF: `osf.io/xxxxx`

4. **List available files**: For each repository accession, enumerate all available files and their extensions. Present this list to the user clearly:

   ```
   Found 14 files across 2 repositories:

   GEO (GSE123456):
     [1] matrix.h5ad        (2.3 GB)
     [2] metadata.csv       (12 KB)
     [3] raw_counts.tsv.gz  (890 MB)
     [4] barcodes.txt       (44 KB)

   Zenodo (10.5281/zenodo.7654321):
     [5] variants.vcf.gz    (340 MB)
     [6] reference.fasta    (3.1 GB)
     [7] README.md          (8 KB)
   ```

5. **Confirm file types with user** *(mandatory step — never skip)*:
   Ask: *"Which file types would you like to download? Please specify extensions (e.g. `h5ad,vcf,fasta`) or say `all`."*
   Wait for the user's answer before proceeding.

6. **Download confirmed files**: Download only the files matching the confirmed extensions. Use streaming downloads with `tqdm` progress bars. Validate MD5/SHA256 checksums where repositories provide them.

7. **Write manifest**: Save `manifest.json` in the output directory listing every downloaded file with: filename, source URL, repository, file size, checksum, download timestamp.

8. **Write report**: Save `report.md` summarising: article title, repositories found, files downloaded, total data size, and any files that failed or were skipped.

**Freedom level:**
- Steps 1–3 (resolution and discovery): **prescriptive** — exact API calls, exact accession pattern matching
- Step 4–5 (listing and confirmation): **prescriptive** — always show the list, always ask
- Step 6 (download): **prescriptive** — never download without confirmation, always validate checksums when available
- Step 8 (report narrative): **flexible** — compose a readable summary

## Supported Repositories

| Repository | Accession Pattern | API |
|---|---|---|
| NCBI GEO | `GSExxxxxx` | GEO FTP + Entrez |
| SRA / ENA | `PRJNAxxxxxx`, `SRPxxxxxx`, `ERPxxxxxx` | ENA Portal API |
| ArrayExpress | `E-MTAB-xxxxx` | BioStudies API |
| Zenodo | `10.5281/zenodo.*` | Zenodo REST API |
| Figshare | `10.6084/*` | Figshare API |
| Dryad | `10.5061/*` | Dryad API |
| OSF | `osf.io/*` | OSF API |

## Supported File Types

The skill can filter for any of these extensions:

| Category | Extensions |
|---|---|
| Genomic variants | `.vcf`, `.vcf.gz`, `.bcf` |
| Sequences | `.fasta`, `.fa`, `.fna`, `.fastq`, `.fastq.gz` |
| Alignments | `.bam`, `.bam.bai`, `.cram` |
| Single-cell | `.h5ad`, `.h5`, `.loom` |
| Tabular | `.csv`, `.tsv`, `.txt`, `.xlsx` |
| Structured data | `.json`, `.yaml` |
| Genomic intervals | `.bed`, `.gff`, `.gtf` |
| Archives | `.gz`, `.zip`, `.tar.gz` |
| Matrix Market | `.mtx`, `.mtx.gz` |

## CLI Reference

```bash
# Standard usage
python skills/article-data-fetcher/article_data_fetcher.py \
  --id 10.1038/s41586-021-03819-2 \
  --types vcf,fasta \
  --output ./downloads

# Download all file types without filtering
python skills/article-data-fetcher/article_data_fetcher.py \
  --id 34613072 \
  --types all \
  --output ./downloads

# Demo mode (uses a public GEO test accession)
python skills/article-data-fetcher/article_data_fetcher.py --demo --output /tmp/demo

# Via ClawBio runner
python clawbio.py run article-data-fetcher --id 10.xxxx/xxxxx --types h5ad,csv --output ./data
```

## Demo

```bash
python clawbio.py run article-data-fetcher --demo
```

Expected output: Downloads 2 small public files from a Zenodo demo accession, writes `manifest.json` and `report.md` to `/tmp/demo`.

## Example Queries

- "Download the VCF and FASTA files from DOI 10.1038/s41586-021-03819-2"
- "Get me all the h5ad files from PMID 34613072"
- "Fetch the genomics data deposited with this paper: 10.1016/j.cell.2022.01.015 — I need CSV and JSON"
- "Download everything from GSE145926"
- "Get the raw counts matrix and metadata from this scRNA-seq paper"

## Example Output

```
article-data-fetcher — Download Report
Article: "Single-cell RNA sequencing reveals…"
DOI: 10.1038/s41586-021-03819-2
Date: 2026-04-23

Repositories found: GEO (GSE123456), Zenodo (10.5281/zenodo.7654321)

Files downloaded (user selected: h5ad, csv):
  ✅ matrix.h5ad         2.3 GB   GSE123456  md5:a1b2c3…
  ✅ metadata.csv        12 KB    GSE123456  md5:d4e5f6…

Files skipped (not in selected types):
  ⏭  raw_counts.tsv.gz  890 MB
  ⏭  variants.vcf.gz    340 MB
  ⏭  reference.fasta    3.1 GB

Total downloaded: 2.3 GB in 2 files
Output directory: ./downloads/GSE123456/

*ClawBio is a research tool. Verify data integrity before use in analysis.*
```

## Output Structure

```
output_dir/
├── report.md
├── manifest.json
└── <accession>/
    ├── matrix.h5ad
    └── metadata.csv
```

`manifest.json` schema:
```json
{
  "article": "10.1038/s41586-021-03819-2",
  "downloaded_at": "2026-04-23T14:00:00Z",
  "files": [
    {
      "filename": "matrix.h5ad",
      "source_url": "https://ftp.ncbi.nlm.nih.gov/geo/series/...",
      "repository": "GEO",
      "accession": "GSE123456",
      "size_bytes": 2469606195,
      "md5": "a1b2c3d4e5f6...",
      "downloaded": true
    }
  ]
}
```

## Dependencies

**Required:**
- `requests>=2.31` — HTTP downloads and API calls
- `tqdm>=4.66` — Progress bars for large file downloads
- `pydantic>=2.0` — Input validation and manifest schema
- `biopython>=1.83` — FASTA/FASTQ parsing for integr