O

Crawlee Python

VERIFIED

Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Parsel, BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.

Downloads 0
Stars 9.1K
Version 1.0.0
Quality 100/100 · Excellent

Install with one command

$ npx skills add apify/crawlee-python

Best for

Web scraping

Find skills for crawling websites, extracting structured data, monitoring pages, and turning messy web content into agent-ready inputs.

Choose it when

  • You want a GitHub-backed skill with 9.1K stars.
  • You need a reusable install command for agents.
  • You want to compare it with related marketplace skills.

Check before install

  • Pushed today
  • License: Apache-2.0
  • Review the repository README and examples.

Quality profile

Excellent candidate for agent workflows

High-confidence pick with strong adoption and healthy maintenance signals.

100
GitHub stars
9.1K
Freshness
Today
Install ready
Yes
License
Apache-2.0

Workflow fit

Use this skill in these scenarios

Collect structured data

Web scraping

I need my agent to scrape websites and extract structured data from pages.

Operate web apps

Browser automation

I need my agent to control a browser, fill forms, and verify web app workflows.

Parse messy files

Document processing

I need my agent to read PDFs, extract tables, and turn documents into structured data.

Stack fit

Add it to a complete workflow

Scrape, clean, and reuse web data

Web data pipeline

A practical stack for agents that crawl public pages, extract clean content, normalize data, and hand it to downstream research or RAG workflows.

Operate and verify web apps

Browser QA agent

A stack for agents that navigate products, fill forms, take screenshots, and verify real user flows across web applications.

Inspect, patch, and verify code

Coding review agent

A stack for software agents that inspect repositories, review pull requests, generate tests, and turn findings into shippable patches.

Overview

Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Parsel, BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.

Imported by the skill-only GitHub discovery pipeline because it matches agent skill, automation, RAG, or developer-tool signals. Protocol-server projects are excluded from automated imports.

Platform Compatibility

pythonFULL
web-automationFULL

Technical Details

Version
1.0.0
License
Apache-2.0
Last Updated
5/23/2026
Published
5/23/2026

Frameworks & Tools

PythonWeb Automation

Install

Free and open source

Compare AlternativesView on GitHubDocumentation

Author

A

apify✓

@apify

Tags

scrapingcrawlerapifyautomationbeautifulsoupcrawlinghacktoberfestheadlessheadless-chromeparsel

Platform Fit

Claude CodeOpenAI AgentsBrowser agents

Health Signals

GitHub stars
9.1K
Quality score
69/100
Last GitHub push
May 22, 2026
Framework hints
2

Community Signal

Share whether this skill looks useful for your agent workflow. Aggregated feedback improves rankings over time.

Trust & Safety

  • —Open source (public GitHub repo)
  • —AI static analysis passed
  • —License: Apache-2.0
  • —Manually verified by team

Related Skills

Crawl4AI

Open-source LLM-friendly web crawler and scraper

66.1K stars · 31.0K installs

Scrapy

High-throughput crawling and scraping for agent data pipelines

61.8K stars · 0 installs

Scrapling

Adaptive web scraping for agent data collection

53.2K stars · 0 installs

EasySpider

A visual no-code/code-free web crawler/spider易采集：一个可视化浏览器自动化测试/数据采集/网页爬虫软件，可以无代码图形化的设计和执行爬虫任务。别名：ServiceWrapper面向Web应用的智能化服务封装系统。

43.9K stars · 0 installs