Build a Dataset from a Queue, Not a Loop

Where this fits

You have a list of 500 URLs and need a consistent set of fields from every page—titles, prices, dates, links. Looping through them one by one is slow, brittle, and hard to resume. The build-a-dataset skill solves that by replacing the loop with a queued crawl: map the site or search the web, filter the URL list, then fetch everything as a batch and poll for the results. It fits at the data-acquisition stage before analysis, indexing, or fine-tuning.

Why agents benefit

• One queued job replaces a fragile for loop over hundreds or thousands of URLs.
• URL discovery and content fetching are separate, so an agent can filter pages before spending credits.
• Map endpoint returns URLs without fetching content; search endpoint returns already scraped results.
• Polling one job is simpler to implement and reason about than managing many concurrent requests.
• The output is a table with the same fields per row, ready for downstream processing.

Practical scenarios

Assembling a full product catalog

An agent needs every product name, price, and availability from an e-commerce site. It calls web/map with a site URL and a limit, filters out category pages, then queues the remaining product URLs for batch fetch.

Building a corpus for fine-tuning

A developer wants a clean dataset of documentation pages. The agent searches for site-specific content, collects the relevant URLs from the search results, and enqueues them in one batch. The returned table becomes the raw corpus.

Enriching external links in a report

An agent has a report full of links and must add the page title and publication date for each. Instead of fetching each link one at a time, it sends the whole known list to the batch endpoint and polls once.

Add it to your agent workflow

Install the skill with:

npx skills add Hydrafetch/skills --skill build-a-dataset

After install, your agent can invoke the skill to map a site, filter URLs, and enqueue a batch fetch. The skill uses Hydrafetch's async job API, so the agent makes one call and then polls until the table is ready. This is far simpler than managing a loop in code.

Compare before adopting

Look at how fresh and complete the returned data is; that is the real quality signal. Compare against alternatives like writing your own scraper or using a generic HTTP client. Check the skill's maintenance freshness on GitHub and whether the author is responsive. Finally, consider workflow fit: does your agent already use npx skills and handle job polling? If not, the pattern may still be worth adopting because queue-based fetching is reusable.

Why it is worth tracking

This skill has zero stars today, so there is no community momentum yet. What makes it worth tracking is the design: queued crawl and batch jobs are a more scalable pattern than loops for agentic data collection. Watch for new commits and usage examples. For a catalog of similar agent skills, see https://www.openagentskill.com/skills/hydrafetch-build-a-dataset. Evaluate it again once your agent needs to build datasets from more than a handful of URLs.