scraping
Use when scraping, extracting, or reading content or data from a website with zendriver — especially JS-rendered SPAs, bot-walled pages, or anything plain fetch/WebFetch can't reach. Covers the open→goto→wait→extract flow, surgical extraction, tool selection, and gotchas.
TurtIeSocks · 已有静态检查记录
安装与使用
运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。
npx skills add TurtIeSocks/zendriver-rs --skill scraping技能说明
name: scraping description: Use when scraping, extracting, or reading content or data from a website with zendriver — especially JS-rendered SPAs, bot-walled pages, or anything plain fetch/WebFetch can't reach. Covers the open→goto→wait→extract flow, surgical extraction, tool selection, and gotchas.
Scraping with zendriver
The canonical playbook for driving the zendriver browser well. The /zendriver:scrape
command and the zendriver-scraper subagent both follow this skill — it is the single
source of truth.
When zendriver is the right tool
Reach for zendriver when the page is JS-rendered, bot-walled, or needs interaction/session state. For static HTML a plain fetch is lighter — don't spin up a browser to read a sitemap.
Core flow
browser_status— is a browser already open this session? Reuse it; don't re-open.browser_open— launch once. Stealth is on by default; do NOT re-enable it or set a profile unless a site specifically needs it.browser_goto { url, wait_for: "load" }— navigate.- Extract (see below).
- Close with
browser_closewhen the job is done (the server also kills its owned Chrome on exit).
The effectiveness lever: surgical extraction, not DOM-dumping
browser_html { trim: true } returns the whole page — ~40–50k tokens for a real article.
Only dump full DOM when you genuinely need the whole page. When the goal is specific,
pull just that:
- A few known elements / text →
browser_find(one) orbrowser_find_all(many). - All links →
browser_get_links. - Structured data, tables, computed values, JSON embedded in
<script>→browser_evaluatewith a small JS snippet that returns exactly the shape you want. - "What resources/endpoints did this page load?" →
browser_search_resources.
Default to targeted tools; reserve browser_html { trim: true } for genuinely unstructured
full-page reads.
Gotchas (hard-won)
wait_for: "load", not"idle"on ad/tracker-heavy sites — idle's ~5s cap times out waiting on hanging tracker requests. Useloadand read the DOM regardless; escalate tobrowser_wait_for_idleonly for SPAs that genuinely settle.- Always
trim: trueonbrowser_htmlunless you need raw markup. - Check
browser_statusbefore assuming a browser/tab is open — calling navigation on a closed browser errors. - A page that loaded but shows little content is often JS-gated — give it a
browser_wait_for_load/ shortbrowser_wait_for_idle, or check for a wall (next section).
Wall detection → hand off
If the DOM shows an anti-bot interstitial instead of content — "Just a moment…", a Cloudflare
ray ID, an Imperva/Incapsula notice, or a DataDome block — invoke the bypass skill
(via the Skill tool) and follow it, then resume extraction.
Multi-page / pagination
Loop the core flow per URL. Prefer browser_get_links to discover "next" targets over
guessing URL patterns. For big crawls, run inside the zendriver-scraper subagent (and
background it) so the main thread's context stays clean.
Responsible use
zendriver is for authorized access: your own sites, content you're permitted to read,
authorized testing, and research. Respect robots-style intent and rate limits; throttle
between requests. Don't use it for mass-targeting or to defeat access controls you have no
right to bypass.