跳至正文
OpenAgentSkill
浏览已开放 · 目录记录日期 2026-11-21 · 自动更新和账户服务暂时关闭。
浏览器与自动化

url

Fetch and extract text from web URLs

axoviq.com · 已有 AI 审核记录

安装与使用

运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。

npx skills add axoviq-ai/synthadoc --skill url

技能说明


name: url version: "1.0" description: Fetch and extract text from web URLs entry: script: scripts/main.py class: UrlSkill triggers: extensions: - "https://" - "http://" intents: - "fetch url" - "web page" - "website" requires:

  • httpx
  • beautifulsoup4 author: axoviq.com license: AGPL-3.0-or-later

URL Skill

Fetches a web URL using httpx, strips navigation/script/style tags with BeautifulSoup, and returns clean body text. PDF URLs are extracted with pypdf (primary) and pdfminer.six (fallback).

Setup

pip install httpx beautifulsoup4

# Optional — needed only if you ingest PDF URLs:
pip install pypdf pdfminer.six

Standalone usage

import asyncio
from synthadoc.skills.url.scripts.main import UrlSkill

skill = UrlSkill()

async def main():
    result = await skill.extract("https://example.com/article")
    print(result.text)          # clean body text
    print(result.metadata)      # {"url": "https://..."}

asyncio.run(main())

DomainBlockedException is raised when the site returns HTTP 401, 403, or 429. Catch it to log and skip the domain:

from synthadoc.skills.base import DomainBlockedException

try:
    result = await skill.extract(url)
except DomainBlockedException as e:
    print(f"Blocked: {e.domain} (HTTP {e.status_code})")

When this skill is used

  • Source starts with https:// or http://
  • User intent contains: fetch url, web page, website