Skip to content
OpenAgentSkill
Browsing is open · Catalog captured 2026-11-21 · Automatic updates and account services are paused.
Development

agentic-browser-testing

Goal-driven E2E testing where a browser agent (Playwright MCP / computer-use) reads a natural-language goal and explores the app via the accessibility tree to assert outcomes — no pre-written script. Covers when intent-driven beats scripted, making agent runs deterministic (pinned model, temperature 0, seeded data, bounded steps, explicit success assertion, snapshot-not-pixel), cost/latency control, the accessibility-tree-first interaction model, CI gating, and graduating a stable run into a scripted Playwright test. Use when: "agentic browser test," "goal-driven browser test," "let an agent explore the app," "natural-language E2E," "browser agent smoke test," "Playwright MCP test." Not for: Writing/maintaining deterministic scripted Playwright tests — that is playwright-automation. Testing your product's OWN LLM features — that is ai-system-testing. Related: playwright-automation, ai-system-testing, exploratory-testing, test-reliability, qa-project-context.

petrkindlmann · AI review on file

Installation

Read the source and dependencies before running a command. Copying does not install or execute anything.

npx skills add petrkindlmann/qa-skills --skill agentic-browser-testing

About this skill


name: agentic-browser-testing description: >- Goal-driven E2E testing where a browser agent (Playwright MCP / computer-use) reads a natural-language goal and explores the app via the accessibility tree to assert outcomes — no pre-written script. Covers when intent-driven beats scripted, making agent runs deterministic (pinned model, temperature 0, seeded data, bounded steps, explicit success assertion, snapshot-not-pixel), cost/latency control, the accessibility-tree-first interaction model, CI gating, and graduating a stable run into a scripted Playwright test. Use when: "agentic browser test," "goal-driven browser test," "let an agent explore the app," "natural-language E2E," "browser agent smoke test," "Playwright MCP test." Not for: Writing/maintaining deterministic scripted Playwright tests — that is playwright-automation. Testing your product's OWN LLM features — that is ai-system-testing. Related: playwright-automation, ai-system-testing, exploratory-testing, test-reliability, qa-project-context. license: MIT metadata: author: kindlmann version: "1.0" category: ai-qa

Quick Route

SituationGo to
Stand up a goal-driven run from scratchDiscovery + references/setup.md
Decide agentic vs scripted for a given flowFit: Intent-Driven vs Scripted
Agent passes one run, fails the nextDeterminism
"How does it click without screenshots?"Interaction Model
Runs are slow / burning tokensCost and Latency
Agent reports false successSuccess Assertion (the Oracle)
Flow is stable — make it permanentGraduation → references/graduation-and-ci.md
Block a merge on the goalCI Gating → references/graduation-and-ci.md
Canvas / no accessibility treeCanvas Fallback → references/graduation-and-ci.md

Discovery Questions

First, check .agents/qa-project-context.md in the project root and skip anything it already answers (stack, environments, seed/reset tooling, model access).

  1. Which flow, and how often does its UI change? Fast-changing/experimental UI favors intent-driven; a stable critical path (login) favors scripted. This decides the whole approach.
  2. Is there a seeded fixture and a way to reset state? Determinism is impossible without seeded data and a per-run reset. If neither exists, that is step zero.
  3. Can you deep-link past auth to a seeded entry point? Re-driving login every run is the biggest avoidable cost; a seeded entry URL scopes the goal and cuts steps.
  4. What is the unambiguous success oracle? Specific account text, a /dashboard URL, an order number — plus a forbidden state. "No error" is not an oracle.
  5. Does the target render to canvas / WebGL? No accessibility tree means snapshot-first won't work; plan the vision fallback or instrument the canvas with ARIA.
  6. Which model and budget? Pin a model id and a step budget up front; tier cheap steps to Haiku 4.5 / Sonnet 4.6 and reserve Opus 4.8 for genuinely ambiguous flows.

Core Principles

  1. Intent, not instructions — but only where churn earns it. The agent reads a goal and finds its own path through the accessibility tree, so it survives a moved button or renamed class that would break a selector. That resilience costs 2-5x the time and money of a scripted run, so spend it on fast-changing UI and hard-to-locate flows, not on stable critical paths.

  2. An agent run is untrustworthy until it is deterministic. Same goal, same seeded app must produce the same verdict. That requires temperature 0, a pinned model id, seeded data with a reset, a bounded step budget, and an explicit pass/fail assertion. Without these you have a coin flip, not a test.

  3. The oracle lives outside the agent. Never let the LLM self-grade "looks good." Success is a checkable assertion against the final browser_snapshot — specific expected text, a URL, AND a forbidden-state negative check — evaluated by your harness, not the model.

  4. Accessibility tree first, pixels last. browser_snapshot returns roles, refs, and accessible names (~200-400 tokens) and is deterministic and cheap. Screenshots, pixel coordinates, vision, and OCR are a scoped last resort for canvas only, never the default.

  5. Graduation is the goal, not perpetual agent runs. Once a flow is stable, promote the run to a durable scripted tests/*.spec.ts with role-based locators. An agent that has been green for two weeks should become a fast, free regression test — keep the agent for exploration, not for guarding a settled path.


Fit: Intent-Driven vs Scripted

The decision is per-flow, not per-project. Run risk-based-testing first if you need the risk map; this table is the routing rule once you have it.

Flow characteristicUseWhy
Stable, high-frequency critical path (login, payment)Scripted + pinned (playwright-automation)Runs every PR; must be fast, free, and deterministic. No upside to re-exploring it.
Fast-changing / experimental UI (a dashboard that churns weekly, a redesign in flight)Agentic / intent-drivenSelectors would break constantly; a goal
Read the complete source ↗