跳至正文
OpenAgentSkill
浏览已开放 · 目录记录日期 2026-11-21 · 自动更新和账户服务暂时关闭。
浏览器与自动化

automation-standards

Scraping + AI-provider standards — the Scraper trait & SCRAPERS registry, selector resilience, rate-limiting/cancellation, and the provider-abstraction (zero-change) rule for embeddings/streaming/prompts. Load for changes under scraping/, ai_provider/, packages/prompts, documents/embed.

saeedkolivand · 已有静态检查记录

安装与使用

运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。

npx skills add saeedkolivand/ai-job-hunter-app --skill automation-standards

技能说明


name: automation-standards description: Scraping + AI-provider standards — the Scraper trait & SCRAPERS registry, selector resilience, rate-limiting/cancellation, and the provider-abstraction (zero-change) rule for embeddings/streaming/prompts. Load for changes under scraping/, ai_provider/, packages/prompts, documents/embed.

Automation standards (scraping + AI-provider)

Authoritative: docs/knowledge/automation-domain.md.

Scraping

  • Register via the registry — scraping/boards/mod.rs (SCRAPERS, Scraper trait, ScraperMode Http/Browser). Don't special-case boards outside the registry.
  • Selector resilience — core boards need fallback selectors; a brittle single-selector parse on a core board is HIGH.
  • Reliability — honor the cancellation token in ScrapeContext; bounded retries with backoff; per-board rate limits; graceful failure recovery (don't poison the queue).
  • Sessions/cookies — handled safely; never log credentials/cookies (security lens → tauri-security-reviewer).

AI provider (the architectural rule — HIGH if violated)

  • No business logic depends on provider-specific APIs. All providers implement a shared interface; adding OpenAI/Anthropic/Gemini/Ollama/OpenRouter/LM Studio = config + adapter only.
  • Embeddings — versioned; on model/space change, invalidation must run (stale embeddings are HIGH).
  • Streaming — partial responses + cancellation handled; no leaks on cancel.
  • Prompts (packages/prompts) — provider-aware + locale-driven, pure TS, zero deps; reusable/composable templates.
  • Cost — minimize token/context; pick the cheapest viable model.

External standards & best-practices (verified 2026-06-19)

AI / LLM (cross-provider — Claude specifics live in the claude-api skill)

  • OWASP Top 10 for LLM Apps (2025) — https://genai.owasp.org/llm-top-10/
    • LLM01 Prompt Injection (direct + indirect) — segregate/label untrusted external content (scraped JD/résumé text is untrusted; never concatenate raw into the instruction block); constrain model role; deterministically validate output; least-privilege tool tokens; human-in-the-loop on high-risk actions. No fool-proof fix → layer defenses.
    • LLM05 Improper Output Handling — treat output as untrusted; parse/validate (Zod/serde) before it reaches IPC/files/render; never eval/shell/SQL with raw output.
    • LLM02 Sensitive-Info Disclosure + LLM07 System-Prompt Leakage — strip PII/secrets from prompts + logs; no secrets in system prompts. LLM06 Excessive Agency — minimal tools/permissions; gate side-effects behind user confirmation.
  • Structured output — prefer native tool/function-calling with constrained decoding over free-text JSON; schemas guarantee shape, not values — still validate.
  • Streaming — SSE is the cross-provider standard; handle partial/aborted streams, disable proxy buffering, support cancellation, prefer partial+error over silent regen.
  • Retries/idempotency — backoff + jitter; retry only idempotent reads; idempotency key/ledger for mutating tool calls.
  • Cost/caching — static prefix first (system prompt + tool schemas) to maximize prompt-cache hits; instrument hit-rate.
  • Evals — version prompts; rerun a fixed eval set on every prompt/model/provider change (2026 models ship faster than your releases and silently shift behavior).

Scraping — legality & resilience

Scope: scrape public, logged-out job postings (listings/JD text), not candidate PII or auth-walled pages. Stay in that lane.

  • Public + logged-out only. Public-data scraping isn't "unauthorized access" under the CFAA (hiQ; narrowed by Van Buren). Never log in, bypass CAPTCHAs, rotate IPs to evade blocks, or defeat auth — that flips CFAA + breach risk. https://www.quinnemanuel.com/the-firm/news-events/client-alert-meta-v-bright-data-significant-decision-for-web-scraping-industry/
  • ⚠️ ToS/contract is the real exposure. Meta v. Bright Data (Jan 2024): logged-OUT scrapers aren't ToS-bound; logged-in scraping stays bound. But 2025 LinkedIn v. Proxycurl etc. show boards winning on contract — treat LinkedIn/Indeed-class sites with named anti-scraping ToS as registry-gated, conservative.
  • ⚠️ EU/GDPR — "public" ≠ free to process personal data; needs lawful basis + the EDPB Opinion 28/2024 three-step legitimate-interest test (+ respect robots.txt, exclude sensitive data, minimize). Clearview fines show "public" is no shield. https://iapp.org/news/a/the-state-of-web-scraping-in-the-eu
  • robots.txt (RFC 9309) — obey it; on 5xx/unreachable assume disallow; cache ≤24h. Identify honestly (stable descriptive UA, no browser spoofing to evade). Rate-limit + backoff per host; honor Retry-After/429. https://www.rfc-editor.org/rfc/rfc9309.html
  • Resilience — semantic selectors (ARIA roles, data-*, JSON-LD JobPosting) over brittle CSS/xpath; version-aware fallback chains; detect drift + fail loudly (no silent empty results); each board isolated in the SCRAPERS registry. Accept anti-bot reality: back off / disable a scraper rather than fingerprint-spoof (spoofing is the legal red line).

Common mistakes: concatenating scraped/untrusted text into the instruction block (LLM01); trusting structured-output values because the shape validated; proxy buffering killing streaming; retrying non-idempotent tool calls; secrets in system prompts; shipping a prompt change with no eval gate; logging in / bypassing CAPTCHA-or-rate-limits to scrape; treating "public" EU personal data as free; brittle selectors with no fallback/drift alarm.