agentic-browser-testing
Goal-driven E2E testing where a browser agent (Playwright MCP / computer-use) reads a natural-language goal and explores the app via the accessibility tree to assert outcomes — no pre-written script. Covers when intent-driven beats scripted, making agent runs deterministic (pinned model, temperature 0, seeded data, bounded steps, explicit success assertion, snapshot-not-pixel), cost/latency control, the accessibility-tree-first interaction model, CI gating, and graduating a stable run into a scripted Playwright test. Use when: "agentic browser test," "goal-driven browser test," "let an agent explore the app," "natural-language E2E," "browser agent smoke test," "Playwright MCP test." Not for: Writing/maintaining deterministic scripted Playwright tests — that is playwright-automation. Testing your product's OWN LLM features — that is ai-system-testing. Related: playwright-automation, ai-system-testing, exploratory-testing, test-reliability, qa-project-context.
petrkindlmann · AI review on file
Installation
Read the source and dependencies before running a command. Copying does not install or execute anything.
npx skills add petrkindlmann/qa-skills --skill agentic-browser-testingAbout this skill
name: agentic-browser-testing description: >- Goal-driven E2E testing where a browser agent (Playwright MCP / computer-use) reads a natural-language goal and explores the app via the accessibility tree to assert outcomes — no pre-written script. Covers when intent-driven beats scripted, making agent runs deterministic (pinned model, temperature 0, seeded data, bounded steps, explicit success assertion, snapshot-not-pixel), cost/latency control, the accessibility-tree-first interaction model, CI gating, and graduating a stable run into a scripted Playwright test. Use when: "agentic browser test," "goal-driven browser test," "let an agent explore the app," "natural-language E2E," "browser agent smoke test," "Playwright MCP test." Not for: Writing/maintaining deterministic scripted Playwright tests — that is playwright-automation. Testing your product's OWN LLM features — that is ai-system-testing. Related: playwright-automation, ai-system-testing, exploratory-testing, test-reliability, qa-project-context. license: MIT metadata: author: kindlmann version: "1.0" category: ai-qa
Quick Route
| Situation | Go to |
|---|---|
| Stand up a goal-driven run from scratch | Discovery + references/setup.md |
| Decide agentic vs scripted for a given flow | Fit: Intent-Driven vs Scripted |
| Agent passes one run, fails the next | Determinism |
| "How does it click without screenshots?" | Interaction Model |
| Runs are slow / burning tokens | Cost and Latency |
| Agent reports false success | Success Assertion (the Oracle) |
| Flow is stable — make it permanent | Graduation → references/graduation-and-ci.md |
| Block a merge on the goal | CI Gating → references/graduation-and-ci.md |
| Canvas / no accessibility tree | Canvas Fallback → references/graduation-and-ci.md |
Discovery Questions
First, check .agents/qa-project-context.md in the project root and skip anything it already
answers (stack, environments, seed/reset tooling, model access).
- Which flow, and how often does its UI change? Fast-changing/experimental UI favors intent-driven; a stable critical path (login) favors scripted. This decides the whole approach.
- Is there a seeded fixture and a way to reset state? Determinism is impossible without seeded data and a per-run reset. If neither exists, that is step zero.
- Can you deep-link past auth to a seeded entry point? Re-driving login every run is the biggest avoidable cost; a seeded entry URL scopes the goal and cuts steps.
- What is the unambiguous success oracle? Specific account text, a
/dashboardURL, an order number — plus a forbidden state. "No error" is not an oracle. - Does the target render to canvas / WebGL? No accessibility tree means snapshot-first won't work; plan the vision fallback or instrument the canvas with ARIA.
- Which model and budget? Pin a model id and a step budget up front; tier cheap steps to Haiku 4.5 / Sonnet 4.6 and reserve Opus 4.8 for genuinely ambiguous flows.
Core Principles
-
Intent, not instructions — but only where churn earns it. The agent reads a goal and finds its own path through the accessibility tree, so it survives a moved button or renamed class that would break a selector. That resilience costs 2-5x the time and money of a scripted run, so spend it on fast-changing UI and hard-to-locate flows, not on stable critical paths.
-
An agent run is untrustworthy until it is deterministic. Same goal, same seeded app must produce the same verdict. That requires temperature 0, a pinned model id, seeded data with a reset, a bounded step budget, and an explicit pass/fail assertion. Without these you have a coin flip, not a test.
-
The oracle lives outside the agent. Never let the LLM self-grade "looks good." Success is a checkable assertion against the final
browser_snapshot— specific expected text, a URL, AND a forbidden-state negative check — evaluated by your harness, not the model. -
Accessibility tree first, pixels last.
browser_snapshotreturns roles, refs, and accessible names (~200-400 tokens) and is deterministic and cheap. Screenshots, pixel coordinates, vision, and OCR are a scoped last resort for canvas only, never the default. -
Graduation is the goal, not perpetual agent runs. Once a flow is stable, promote the run to a durable scripted
tests/*.spec.tswith role-based locators. An agent that has been green for two weeks should become a fast, free regression test — keep the agent for exploration, not for guarding a settled path.
Fit: Intent-Driven vs Scripted
The decision is per-flow, not per-project. Run risk-based-testing first if you need the
risk map; this table is the routing rule once you have it.
| Flow characteristic | Use | Why |
|---|---|---|
| Stable, high-frequency critical path (login, payment) | Scripted + pinned (playwright-automation) | Runs every PR; must be fast, free, and deterministic. No upside to re-exploring it. |
| Fast-changing / experimental UI (a dashboard that churns weekly, a redesign in flight) | Agentic / intent-driven | Selectors would break constantly; a goal |