ai-test-generation
Use AI to write NEW test code from specs, PRDs, user stories, code diffs, bug reports, or OpenAPI specs. Staged pipeline: requirements extraction → risk analysis → coverage matrix → scenario generation → oracle design → test code → human review, with guardrails against hallucinated APIs and weak assertions. Use when: "generate tests from spec," "tests from PRD," "tests from user story," "auto-generate test cases," "AI write tests for me." Not for: testing AI/LLM features in your product — use ai-system-testing. Not for: auditing a pre-existing test suite you did not just generate — use ai-qa-review (Step 7 here only reviews tests THIS pipeline produced). Related: playwright-automation, unit-testing, api-testing, qa-project-context.
petrkindlmann · 已有 AI 审核记录
安装与使用
运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。
npx skills add petrkindlmann/qa-skills --skill ai-test-generation技能说明
name: ai-test-generation description: >- Use AI to write NEW test code from specs, PRDs, user stories, code diffs, bug reports, or OpenAPI specs. Staged pipeline: requirements extraction → risk analysis → coverage matrix → scenario generation → oracle design → test code → human review, with guardrails against hallucinated APIs and weak assertions. Use when: "generate tests from spec," "tests from PRD," "tests from user story," "auto-generate test cases," "AI write tests for me." Not for: testing AI/LLM features in your product — use ai-system-testing. Not for: auditing a pre-existing test suite you did not just generate — use ai-qa-review (Step 7 here only reviews tests THIS pipeline produced). Related: playwright-automation, unit-testing, api-testing, qa-project-context. license: MIT metadata: author: kindlmann version: "2.0" category: ai-qa
Before starting: Check for .agents/qa-project-context.md in the project root. It carries tech stack, test frameworks, naming conventions, selector strategy, and known risk areas that dramatically improve generated test quality.
Quick Route
The pipeline is the same for every input; only the Step 1 extraction emphasis changes. Jump to the matching row, then run Steps 2-7 unchanged.
| Input type | Step 1 extracts | Watch for |
|---|---|---|
| PRD / feature spec | Entities, business rules, acceptance criteria, NFRs, stated assumptions | Implicit requirements inferred from "seamless"/"fast" language |
| User story + AC | Each AC → ≥1 happy + ≥1 negative scenario | ACs that hide multiple behaviors in one line |
Code diff (git diff main...HEAD) | New/changed code paths, modified conditionals, removed behavior | Regression scope: test the changed paths, not the whole module |
| Bug report | Repro steps, expected vs actual, environment | Write a test asserting expected — fails now, passes after fix |
| OpenAPI / GraphQL SDL | Endpoints, schemas, required fields, enums, auth | Validation, auth-failure, and edge cases per endpoint, not just 200s |
Playwright projects also pick an agent integration mode — see Discovery Q2.
Discovery Questions
Check .agents/qa-project-context.md first — if it exists, use it and skip anything already answered there. Then clarify:
-
What is the input source? PRD / spec, user story + AC, code diff, bug report, or API schema. Determines Step 1 extraction emphasis (see Quick Route). For an LLM/AI feature spec, stop — generate eval datasets in
ai-system-testing, not Playwright specs. -
What is the target test framework, and (for Playwright) which agent integration mode?
- E2E: Playwright (preferred), Cypress. Unit: Jest, Vitest, pytest. API: Playwright
APIRequestContext, Supertest, requests. - Playwright CLI + agents (recommended for Claude Code / Codex / Cursor):
npx playwright init-agents --loop=claudescaffolds planner/generator/healer agents into.claude/agents/as markdown. They are interactive dev tools that produce standard Playwright tests which run unchanged in CI. Token-efficient; runs inside the agent's loop. - Playwright MCP (
npx @playwright/mcp@latest): higher overhead, right when the agent must drive a live browser interactively over a long session. - Neither — hand-write tests using AI as a scratch-pad helper.
- E2E: Playwright (preferred), Cypress. Unit: Jest, Vitest, pytest. API: Playwright
-
What project context is available? Existing test patterns, Page Objects / helpers, data factories / fixtures, CI constraints (timeout, parallelism). More context = less cleanup.
-
What is the review workflow? Full pipeline → human review → merge (default); scenarios only → human writes code; or code → human refines iteratively.
-
What domain knowledge is needed? Regulated industry (healthcare, finance) compliance, domain invariants (money never negative, appointments cannot overlap), known risk areas from past incidents.
Core Principles
-
Pipeline before code. Never generate test code before establishing what to test, why, and how to verify it. The seven-step pipeline exists to prevent premature code generation that targets the wrong things.
-
Structured intermediates are the product. The assumptions document, coverage matrix, and oracle definitions are more valuable than the test code itself. They are reviewable, traceable, and reusable.
-
Separate what from how. Scenario generation (what to test) and oracle design (how to verify) are distinct cognitive tasks. Mixing them produces scenarios biased toward what is easy to assert, with assertions tacked on as afterthoughts.
-
AI generates the first draft; a human reviews and refines. Never ship AI-generated tests without human review. The AI accelerates — it does not replace judgment.
-
Context is everything. Feed the LLM your conventions, existing patterns, selector strategy, and data setup. The more context, the less cleanup.
-
Quality over quantity. Each test has a maintenance cost. Focus on critical paths, complex logic, and known risk areas — not test count.
The Pipeline
Mandatory workflow — agents MUST follow this order:
Step 1: Extract → Requirements, entities, business rules from input
Step 2: Analyze → Risks, invariants, edge cases, ambiguities
Step 3: Map → Coverage matrix (requirement → scenario → priority)
Step 4: Generate → Candidate scenarios (happy + boundary + negative + security + a11y)
Step 5: Design → Assertions and oracles SEPARATELY from scenarios
Step 6: Code → Test code (only a
阅读完整来源 ↗