cp-skill-snapshot-web
Snapshot a URL into the local kb/sources/.snapshots/ cache, routing GitHub, X/Twitter, PDF, and ordinary web sources to the appropriate capture path.
zby · AI review on file
Installation
Read the source and dependencies before running a command. Copying does not install or execute anything.
npx skills add zby/commonplace --skill cp-skill-snapshot-webAbout this skill
name: cp-skill-snapshot-web description: Snapshot a URL into the local kb/sources/.snapshots/ cache, routing GitHub, X/Twitter, PDF, and ordinary web sources to the appropriate capture path. type: kb/types/instruction.md user-invocable: true allowed-tools: Read, Write, Grep, Glob, Bash context: fork model: sonnet argument-hint: "[url] — URL to snapshot (web page, PDF, GitHub issue/PR, or X/Twitter post)"
EXECUTE NOW
Target: $ARGUMENTS
If no URL provided, ask the user for one.
If URL provided, start Step 1 immediately.
START NOW.
Step 1: Verify Local Storage and Check for Duplicates
Keep the provided URL as source_url. Verify that
kb/sources/.snapshots/ is ignored by the project. The shipped scaffold does
this through kb/sources/.gitignore. If the directory is not ignored, stop
before writing and report the missing rule.
Use Grep to search for an exact frontmatter source: {source_url} in existing
Markdown files in kb/sources/.snapshots/. If found, compute the SHA-256 of
the exact file bytes, tell the user, and stop:
Already snapshotted: kb/sources/.snapshots/{filename} SHA-256: {64-character lowercase checksum}
Step 2: Route by URL Type
Detect the source_url type and branch:
- GitHub issue/PR (
github.com/.../issues/Norgithub.com/.../pull/N) → Step 2a - X/Twitter (
x.com/.../status/...ortwitter.com/.../status/...) → Step 2b - arXiv abstract page (
arxiv.org/abs/...) → Step 2c - PDF (URL ends in
.pdf, orarxiv.org/pdf/) → Step 2c - Everything else → Step 2d
Step 2a: GitHub Issue/PR
Run:
commonplace-github-snapshot "{source_url}"
Parse either the Snapshot saved: or Already snapshotted: line from the
output to get the file path. Tell the user and stop — the script handles
metadata, formatting, and saving.
Step 2b: X/Twitter Post
Run:
commonplace-x-snapshot "{source_url}"
Parse either the Snapshot saved: or Already snapshotted: line from the
output to get the file path. Tell the user and stop — the script handles
metadata, formatting, and saving.
Step 2c: Resolve and Fetch PDF
Verify that the PDF capture prerequisites are available:
command -v curl
command -v pdfinfo
command -v pdftotext
If any command is missing, go to Step 3. Do not probe for an alternative converter.
Set pdf_url:
- For an arXiv abstract URL, replace
/abs/with/pdf/and discard any query string or fragment. Preserve an explicit terminal version such asv1. If the abstract URL has no terminal version, leave the PDF URL unversioned so arXiv serves the latest paper version. For example,https://arxiv.org/abs/2606.03979becomeshttps://arxiv.org/pdf/2606.03979. Do not route the abstract page through ordinary HTML capture. - For an existing PDF URL, use
source_urlunchanged.
Run this as one Bash invocation. Retain the printed directory path as
{snapshot_tmp}:
set -e
snapshot_tmp=$(mktemp -d)
printf 'Snapshot temp: %s\n' "$snapshot_tmp"
curl -fsSL -o "$snapshot_tmp/source.pdf" "{pdf_url}"
pdfinfo -isodates "$snapshot_tmp/source.pdf" > "$snapshot_tmp/pdfinfo.txt"
pdfinfo -meta "$snapshot_tmp/source.pdf" > "$snapshot_tmp/pdfmeta.xml" || true
pdftotext -enc UTF-8 -eol unix -nopgbrk \
"$snapshot_tmp/source.pdf" "$snapshot_tmp/extracted.txt"
Use Read to inspect pdfinfo.txt, non-empty pdfmeta.xml, and a bounded
beginning of extracted.txt. DOI metadata inspection is best effort and its
failure does not make an otherwise successful capture fail. Treat PDF metadata
fields as leads, not as authority: confirm the title and authors against the
document text when available. Use Grep plus bounded Read ranges to locate an
abstract, executive summary, introduction, or the source document's own DOI
when the beginning does not supply enough metadata. Do not treat a DOI found
only in the references as the paper's DOI. Do not read the whole extracted
file merely to copy it. If extracted.txt is empty or contains no substantive
text, go to Step 3.
Set capture_method to pdftotext, set body_file to
{snapshot_tmp}/extracted.txt, and go to Step 4.
Step 2d: Fetch Web Page
Verify that the HTML capture prerequisites are available:
command -v trafilatura
If the command is missing, go to Step 3. Do not probe for another HTML converter.
Run this as one Bash invocation to download and extract the page. Retain the
printed directory path as {snapshot_tmp}:
set -e
snapshot_tmp=$(mktemp -d)
printf 'Snapshot temp: %s\n' "$snapshot_tmp"
trafilatura -u "{source_url}" \
--markdown --with-metadata --links --no-comments --recall \
--backup-dir "$snapshot_tmp/raw" \
> "$snapshot_tmp/extracted.md"
Use Read to inspect only the leading metadata and a bounded beginning of
extracted.md. Its leading YAML block, when present, is Trafilatura metadata:
retain it as input to Step 4 but do not copy that block into the snapshot body.
Trafilatura also retains its downloaded HTML as a gzip file under
{snapshot_tmp}/raw/. When gzip is available, decompress that file within
{snapshot_tmp} and use Grep with bounded output to inspect article-level DOI
metadata such as citation_doi, dc.identifier, prism.doi, or a JSON-LD
doi property. DOI inspection is best effort: inability to inspect the raw
HTML does not make an otherwise successful capture fail.
Strip that block locally without re-emitting the document:
awk '
NR == 1 && $0 == "---" { in_metadata = 1; next }
in_metadata && $0 == "---" { in_metadata = 0; next }
!in_metadata { print }
' "{snapshot_tmp}/extracted.md" > "{snapshot_tmp}/body.md"
If body.md is empty or contains no substantive main content, go to
Step 3.
Set capture_method to trafilatura, set body_file to
{snapshot_tmp}/body.md, and go to Step 4.
Step 3: Handle Failures
If any fetch or extraction method fails (missing prerequisite, curl e
Read the complete source ↗