research

Fetch Live Websites in AI Agents with moli-webfetch

A skill for Codex to fetch, crawl, and capture JavaScript-rendered web content in agent workflows.

by lexmount853 GitHub stars

Where this fits

You're building an agent that needs to query a live pricing page, read a dynamically loaded article, or follow a series of links—but the content is gated behind client-side JavaScript. Your agent gets a static HTML shell and misses the data. That's where moli-webfetch steps in: it gives your agent a headless browser command that executes scripts, maintains the live DOM, and returns clean output (HTML, Markdown, JSON, even screenshots) without forcing you to manage a browser stack yourself.

Why agents benefit

  • Live DOM by default: Moli runs JavaScript and waits for network idle, so agents see the same content a human would, including infinite-scroll sections or tabs activated by click events.
  • Structured output for agents: Fetch returns semantic trees or Markdown/JSON, making it easy for your agent to extract facts without parsing raw HTML.
  • Bounded crawling: Follow links up to a depth or count you specify, letting agents perform lightweight site exploration without losing focus.
  • Render artifacts on demand: Generate viewport or full-page screenshots, PDFs, or WebPageTest traces when the agent needs visual verification or archival.
  • Zero infrastructure: One-shot binary works on Linux, macOS, and Windows via a curl or PowerShell install—no persistent service required.

Practical scenarios

Scenario 1: Researching a JS‑heavy product page

Your agent is asked, "What are the current shipping options for product X?" The page reveals options only after a script animates a drop-down. Moli fetches the page, executes the script, and returns the rendered text—agent extracts the answer in one step.

Scenario 2: Following a link trail for fact verification

An agent needs to verify a claim across three pages, each linked from the previous one. With a bounded crawl (max pages=5, max depth=2), it can pull all relevant content in a single Moli invocation and cross-check facts without multiple tool calls.

Scenario 3: Finding content inside a password-protected dashboard

If the agent has credentials, it can post a login form, wait for the redirect, then capture the post-auth page as a PDF. Moli's network interception and cookie handling make this feasible without building a separate browser automation script.

Add it to your agent workflow

Install the skill with OpenAgentSkill:

npx skills add lexmount/moli --skill moli-webfetch

Then, inside your agent's tool loop, Moli's fetch command accepts a URL and optional flags. For example, to get a Markdown version of a page:

moli fetch https://example.com --output markdown

Or to capture a full-page screenshot:

moli fetch https://example.com --screenshot full-page

Resolve moli from PATH; the skill itself includes an installer if the binary isn't present. For a deeper look at how this skill fits into the broader agent skill ecosystem, see the OpenAgentSkill listing for moli-webfetch.

Compare before adopting

Before committing, compare against alternatives like Playwright MCP or a simple curl+lynx pipeline. Check how often Moli's prebuilt binaries are updated (look at GitHub releases), whether it handles your target sites' anti-bot measures, and whether its output formats mate well with your agent's existing parsers. Also weigh the cost of a headless browser for tasks that simple HTTP requests can solve—Moli is only worth it when dynamic rendering matters.

Why it is worth tracking

With 853 stars and an active single-binary distribution, Moli shows real community momentum. Its ability to produce a semantic tree or screenshot in a one-off command makes it a strong candidate for agents that need occasional visual or interactive web access. Evaluate it when your agent starts hitting walls with static fetches, or when you need a reliable way to produce page artifacts for debugging and auditing.

Featured Skill

moli-webfetch