Skip to content
OpenAgentSkill
Browsing is open · Catalog captured 2026-11-21 · Automatic updates and account services are paused.
AI & knowledge

webforai

Convert web pages or local HTML files to clean, LLM-ready Markdown with the webforai CLI. Use when a task needs page content as Markdown - reading documentation or articles, scraping, summarising URLs, crawling a docs site into files or llms.txt, building text corpora - including JavaScript-heavy or bot-protected pages via the hosted platform.

inaridiy · Static checks on file

Installation

Read the source and dependencies before running a command. Copying does not install or execute anything.

npx skills add inaridiy/webforai --skill webforai

About this skill


name: webforai description: Convert web pages or local HTML files to clean, LLM-ready Markdown with the webforai CLI. Use when a task needs page content as Markdown - reading documentation or articles, scraping, summarising URLs, crawling a docs site into files or llms.txt, building text corpora - including JavaScript-heavy or bot-protected pages via the hosted platform.

webforai CLI

Convert a URL or local HTML file to Markdown. Non-interactive by default: the Markdown goes to stdout, every log goes to stderr, so output can be piped or captured directly.

Basic usage

npx webforai-cli https://example.com/article            # Markdown to stdout
npx webforai-cli https://example.com/article -o out.md  # write to a file instead
npx webforai-cli ./page.html                            # convert a local HTML file
npx webforai-cli https://example.com --json             # machine-readable envelope (see below)

Which command

  • One page, or a handful you will read right away → npx webforai-cli <url> (free, local). When the page needs JavaScript, add -l playwright (a local Chromium, still free). With WEBFORAI_API_KEY set, --engine auto (hosted platform) also gets past sites that block plain fetches.
  • A known list of URLs (2–100) to keep as files → npx webforai-cli batch (hosted platform).
  • A whole site or docs section → npx webforai-cli crawl (hosted platform); add --llms-txt for a single llms-full.txt you can read in one go.
  • No Node.js, or only an HTTP fetch tool → GET https://platform.webforai.dev/<url> (for example https://platform.webforai.dev/https://example.com/article) returns the page as Markdown, no key needed: a demo limited to 5 requests per 10 minutes and cut at 40,000 characters. With Authorization: Bearer $WEBFORAI_API_KEY it is a normal billed scrape, full length.

Flags

  • -o, --output <path> — write Markdown to a file instead of stdout.
  • -l, --loader <fetch|playwright|platform> — how HTML is acquired. Default: fetch for URLs, automatic local for file paths. playwright renders JavaScript locally (needs npx playwright install chromium). platform uses the hosted API (below).
  • -m, --mode <default|ai> — ai strips links/tables/images down to plain text (local conversion only).
  • --extractor <auto|reader|agent|comments|kiwame|takumi|minimal|none> — main-content extraction preset (agent: main content plus reader comments and role-grouped links; comments: main content plus reader comments) (kiwame is the learned block classifier, takumi the heuristic; none converts the whole page).
  • --frontmatter — prepend YAML front matter (title, author, canonical URL, ...).
  • --json — print a JSON envelope to stdout instead of raw Markdown.
  • --engine <auto|fetch|browser|proxy-fetch|proxy-browser> — platform acquisition engine (implies -l platform). auto (the platform default) starts with a plain fetch and escalates to browser rendering when the page is a client-rendered shell; browser/proxy-browser always render JavaScript; proxy-* engines egress through rotating proxies for bot-protected sites (paid plan only).
  • --region <auto|jp> — platform proxy egress region (implies -l platform); jp runs on the proxy tier, so it needs a paid plan too.
  • --screenshot — platform browser engines only; the envelope carries an expiring screenshotUrl.
  • --respect-robots — platform loader only; honor the target site's robots.txt rules (off by default). A disallowed URL fails with robots_disallowed and is not billed.
  • --api-key <key> / --platform-url <url> — platform credentials and host; usually set via env vars instead.
  • -i, --interactive — guided prompt flow (for humans; do not use from an agent).
  • -d, --debug — verbose logs on stderr.

JSON envelope (--json)

{
  "source": "https://example.com",
  "loader": "fetch",
  "url": "https://example.com",
  "markdown": "# ...",
  "metadata": { "title": "..." },
  "engine": "browser",
  "region": "auto",
  "credits": 5,
  "screenshotUrl": "https://...",
  "output": "out.md"
}

engine/region/credits/screenshotUrl appear only for the platform loader; output only when -o also wrote a file. A warning field appears when the result is probably degraded — most commonly a client-rendered shell fetched without JavaScript; rerun with -l playwright when you see one (or --engine auto when WEBFORAI_API_KEY is set).

Hosted platform

For JavaScript-heavy or bot-protected pages, the hosted platform fetches server-side:

export WEBFORAI_API_KEY=wfa_...       # from https://platform.webforai.dev/dashboard
npx webforai-cli https://example.com --engine auto --json

WEBFORAI_PLATFORM_URL overrides the host for self-hosted deployments. Accounts get 1,000 free credits per month; failures are never billed.

operationcredits
fetch1
browser2
proxy-fetch (paid plan only)2
proxy-browser (paid plan only)3
--screenshot+1
image rehosting (API rehostImages)+1 per started 5 images

auto bills the engine that actually ran (1, or 2 when it needed a browser). The proxy engines — and --region jp, which runs on them — need an active subscription; on the free allowance they fail with payment_required. Prefer --engine auto unless a site blocks it. Batch and crawl jobs bill per converted page.

Crawl and batch (hosted platform)

# a docs site → one .md per page, mirroring URL paths, plus llms.txt / llms-full.txt
npx webforai-cli crawl https://docs.example.com -o docs --limit 100 --include '^/docs' --llms-txt

# a list of URLs (arguments, or one per line via --file; '-' reads stdin)
npx webforai-cli batch https://a.example/post https://b.example/news -o pages
npx webforai-cli batch --file urls.txt -o pages --json

Both submit an async job, wait for it, then write every converted page under `-o <dir

Read the complete source ↗