webforai
Convert web pages or local HTML files to clean, LLM-ready Markdown with the webforai CLI. Use when a task needs page content as Markdown - reading documentation or articles, scraping, summarising URLs, crawling a docs site into files or llms.txt, building text corpora - including JavaScript-heavy or bot-protected pages via the hosted platform.
inaridiy · 已有静态检查记录
安装与使用
运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。
npx skills add inaridiy/webforai --skill webforai技能说明
name: webforai description: Convert web pages or local HTML files to clean, LLM-ready Markdown with the webforai CLI. Use when a task needs page content as Markdown - reading documentation or articles, scraping, summarising URLs, crawling a docs site into files or llms.txt, building text corpora - including JavaScript-heavy or bot-protected pages via the hosted platform.
webforai CLI
Convert a URL or local HTML file to Markdown. Non-interactive by default: the Markdown goes to stdout, every log goes to stderr, so output can be piped or captured directly.
Basic usage
npx webforai-cli https://example.com/article # Markdown to stdout
npx webforai-cli https://example.com/article -o out.md # write to a file instead
npx webforai-cli ./page.html # convert a local HTML file
npx webforai-cli https://example.com --json # machine-readable envelope (see below)
Which command
- One page, or a handful you will read right away →
npx webforai-cli <url>(free, local). When the page needs JavaScript, add-l playwright(a local Chromium, still free). WithWEBFORAI_API_KEYset,--engine auto(hosted platform) also gets past sites that block plain fetches. - A known list of URLs (2–100) to keep as files →
npx webforai-cli batch(hosted platform). - A whole site or docs section →
npx webforai-cli crawl(hosted platform); add--llms-txtfor a singlellms-full.txtyou can read in one go. - No Node.js, or only an HTTP fetch tool →
GET https://platform.webforai.dev/<url>(for examplehttps://platform.webforai.dev/https://example.com/article) returns the page as Markdown, no key needed: a demo limited to 5 requests per 10 minutes and cut at 40,000 characters. WithAuthorization: Bearer $WEBFORAI_API_KEYit is a normal billed scrape, full length.
Flags
-o, --output <path>— write Markdown to a file instead of stdout.-l, --loader <fetch|playwright|platform>— how HTML is acquired. Default:fetchfor URLs, automaticlocalfor file paths.playwrightrenders JavaScript locally (needsnpx playwright install chromium).platformuses the hosted API (below).-m, --mode <default|ai>—aistrips links/tables/images down to plain text (local conversion only).--extractor <auto|reader|agent|comments|kiwame|takumi|minimal|none>— main-content extraction preset (agent: main content plus reader comments and role-grouped links;comments: main content plus reader comments) (kiwameis the learned block classifier,takumithe heuristic;noneconverts the whole page).--frontmatter— prepend YAML front matter (title, author, canonical URL, ...).--json— print a JSON envelope to stdout instead of raw Markdown.--engine <auto|fetch|browser|proxy-fetch|proxy-browser>— platform acquisition engine (implies-l platform).auto(the platform default) starts with a plain fetch and escalates to browser rendering when the page is a client-rendered shell;browser/proxy-browseralways render JavaScript;proxy-*engines egress through rotating proxies for bot-protected sites (paid plan only).--region <auto|jp>— platform proxy egress region (implies-l platform);jpruns on the proxy tier, so it needs a paid plan too.--screenshot— platform browser engines only; the envelope carries an expiringscreenshotUrl.--respect-robots— platform loader only; honor the target site's robots.txt rules (off by default). A disallowed URL fails withrobots_disallowedand is not billed.--api-key <key>/--platform-url <url>— platform credentials and host; usually set via env vars instead.-i, --interactive— guided prompt flow (for humans; do not use from an agent).-d, --debug— verbose logs on stderr.
JSON envelope (--json)
{
"source": "https://example.com",
"loader": "fetch",
"url": "https://example.com",
"markdown": "# ...",
"metadata": { "title": "..." },
"engine": "browser",
"region": "auto",
"credits": 5,
"screenshotUrl": "https://...",
"output": "out.md"
}
engine/region/credits/screenshotUrl appear only for the platform loader; output
only when -o also wrote a file. A warning field appears when the result is probably
degraded — most commonly a client-rendered shell fetched without JavaScript; rerun with
-l playwright when you see one (or --engine auto when WEBFORAI_API_KEY is set).
Hosted platform
For JavaScript-heavy or bot-protected pages, the hosted platform fetches server-side:
export WEBFORAI_API_KEY=wfa_... # from https://platform.webforai.dev/dashboard
npx webforai-cli https://example.com --engine auto --json
WEBFORAI_PLATFORM_URL overrides the host for self-hosted deployments. Accounts get 1,000
free credits per month; failures are never billed.
| operation | credits |
|---|---|
fetch | 1 |
browser | 2 |
proxy-fetch (paid plan only) | 2 |
proxy-browser (paid plan only) | 3 |
--screenshot | +1 |
image rehosting (API rehostImages) | +1 per started 5 images |
auto bills the engine that actually ran (1, or 2 when it needed a browser). The proxy
engines — and --region jp, which runs on them — need an active subscription; on the free
allowance they fail with payment_required. Prefer --engine auto unless a site blocks
it. Batch and crawl jobs bill per converted page.
Crawl and batch (hosted platform)
# a docs site → one .md per page, mirroring URL paths, plus llms.txt / llms-full.txt
npx webforai-cli crawl https://docs.example.com -o docs --limit 100 --include '^/docs' --llms-txt
# a list of URLs (arguments, or one per line via --file; '-' reads stdin)
npx webforai-cli batch https://a.example/post https://b.example/news -o pages
npx webforai-cli batch --file urls.txt -o pages --json
Both submit an async job, wait for it, then write every converted page under `-o <dir
阅读完整来源 ↗