AI Agent Skills

Cross Platform C# web crawler framework built for speed and flexibility. Please star this project! +1.

Polite, slim and concurrent web crawler.

The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns

The only open-source toolkit that can download SEC EDGAR financial reports and extract textual data from specific item sections into nice & clean structured JSON files.…

PHP script to recursively crawl websites and generate a sitemap. Zero dependencies.

A Facebook crawler

Turn websites into clean markdown or structured data for retrieval and agents.

A simple and flexible web crawler that follows the robots.txt policies and crawl delays.

SiteOne Crawler GUI is a cross-platform website crawler and analyzer for SEO, security, accessibility, and performance optimization—ideal for developers, DevOps, QA engi…

🎭 Playwright integration for Scrapy

带你了解一下Golang的市场行情

Web crawling built for AI

Elegant Scraper and Crawler Framework for Golang

Structured data gathering from any website using AI-powered scraper, crawler, and browser automation. Scraping and crawling with natural language prompts. Equip your LLM…

AI crawler access analysis. Checks robots.txt, meta tags, and HTTP headers to determine which AI crawlers can access the site. Provides a complete access map and recomme…

Collect Bilibili video and creator evidence with MediaCrawler through search, exact BV detail, comments, dynamics, contacts, and optional media. Use for topic, format, t…