derive-tests
'What should we test' or a coverage map: derive the coverage map and missing tests from docs and code; check mode reports staleness.
phuryn · Static checks on file
Installation
Read the source and dependencies before running a command. Copying does not install or execute anything.
npx skills add phuryn/product-engineering --skill derive-testsAbout this skill
name: derive-tests description: "'What should we test' or a coverage map: derive the coverage map and missing tests from docs and code; check mode reports staleness." argument-hint: "[check] <repo path or area; defaults to the whole repository>"
Derive tests -- Turn Intent Into Tests
The docs say what the system should do. An audit finds where the code doesn't. Tests are what stop that gap from reopening after the next AI edit. This skill reads the documented intent, turns each load-bearing rule into a concrete test case, sorts them into what to automate, what needs a guarded live run, and what stays manual — then recommends the gate that keeps main honest: the one the repo's own instructions define, or a green-before-merge CI gate where they define none.
This produces a coverage map (tests.md) and concrete test cases, not a finished suite — you or the next agent implement the deterministic ones. If the repo already keeps a test-coverage file, under any name, that file is the map: update it rather than starting a second one (the shipping-artifacts skill's Locations and names rule).
Invocation
/ai-shipping:derive-tests Claude Code
/ai-shipping:derive-tests the checkout flow
$ai-shipping:derive-tests supabase/functions Codex
/ai-shipping:derive-tests check
"what should we test here" · "is our coverage map current" either agent, in plain words
A request that starts with check, or asks only whether the map is current, runs check-only mode (below) on the scope that follows it, instead of the workflow.
Prerequisite: documented intent
Tests are derived from the docs, so the docs come first. If the system docs (in the repo's docs location, or documentation/ by default) are missing or thin, apply the document-app skill first (this one reads flows.md, permissions.md, and automation.md most heavily). You cannot map coverage to rules you never wrote down — where intent is absent, say so rather than inventing rules to test.
The workflow
1. Read the intent — and the tests that already exist
Read the applicable system docs (architecture, flows, permissions, variables, and any of emails, cron, seo, automation that exist). Apply the shipping-artifacts skill for what each doc should contain, and the intended-vs-implemented skill for the discipline of treating docs as claims to verify, not proof.
Then inventory the existing test suite — the test files, what they actually assert, and what runs in CI today. The map you produce must distinguish coverage that exists now from coverage you're proposing; skipping this step yields a falsely-green map that claims rules are pinned when nothing checks them. If there are no tests, say so plainly — that is itself a finding.
2. Extract the rules worth testing
Pull out the load-bearing, deterministic rules — the ones whose violation crosses a trust, data, money, tenant, or privacy boundary:
- authorization allow and deny cases (especially the boundary crossings in
flows.mdand the matrix inpermissions.md), - input validation and output encoding at each sink,
- idempotency of jobs and dedup keys,
- fail-closed defaults (error / timeout / cache-miss / flag paths that must deny, not allow),
- side-effect conditions (exactly when an email sends, a write commits, a paid action fires),
- public-data-only constraints on public or bot routes,
- the output-contract and tool-surface limits of any agent in
automation.md.
Skip cosmetic behavior. A rule earns a test when getting it wrong harms someone other than the actor.
3. Build the coverage map
One row per use case: rule → expected behavior (incl. the negative case) → evidence source (doc + code) → test type → status (existing / proposed / none). The status column is what keeps the map honest — mark a rule existing only when a test in the repo actually asserts it today.
Test types:
- unit — pure and deterministic, no external services.
- integration (deterministic) — exercises real wiring against a dependency the product runs for its tests (a local or development database, a mocked provider) and passes or fails the same way every time.
- guarded live — needs a real external service the product does not run for its tests: an email provider, an LLM, a third party. Runs behind an explicit flag.
- manual — UI/visual or judgment calls. A reviewer checklist item, not an automated test.
What a test depends on and whether it gates are separate properties, and the repo's gate policy decides the second (Step 5). A product whose gate runs against its development database and the identity provider's development instance, with real sessions in a browser (the test-harness skill), gates on those rows: they are deterministic there, because the product owns those environments. Where the repo states no gate, the gate requires the deterministic set — unit plus integration tests that pass or fail the same way on every run — and guarded-live and manual rows don't gate. Either way, prefer unit where the decision logic can be isolated, and reach for integration when the rule lives in the wiring (middleware, RLS, auth guards) and only a real dependency can exercise it.
When a rule can only be exercised live, you can extract its decision into a pure helper so the logic is unit-testable — but only as a complement, not a replacement for testing the real enforcement. The unit test proves the helper's logic; it does not prove the framework actually calls it. Wiring and policy enforcement (route middleware, DB row-level security, auth guards, provider config) still needs an integration or guarded-live check, or the helper becomes a policy shadow that passes while the real path is unprotected.
4. Propose the tests
For each rule you can pin with a deterministic automated test (unit or integration), write the case: name, arrange/act/assert intent, and the negative ca
Read the complete source ↗