audit-prove-it
Try to fool a repo's verification skill, either kind made by /create-prove-it (the permanent prove-<app> or a temporary prove-<migration>). Plants one deliberate bug at a time in the files the current change touched (or along a flow the user names), and for a migration only in the new system, never the old one it is compared with. A fresh agent then checks that change or flow with the skill and its answer keys, without knowing what was planted. Records whether it caught the bug, reverts exactly, and reports every miss with a suggested fix to the skill and to the flow's answer key. Use for /audit-prove-it, "can the verifier be fooled", "test the verify skill", "is our verification any good", "mutation test the verifier", or after creating or heavily editing a prove-* skill.
ray-amjad · 已有 AI 审核记录
安装与使用
运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。
npx skills add ray-amjad/skills --skill audit-prove-it技能说明
name: audit-prove-it description: Try to fool a repo's verification skill, either kind made by /create-prove-it (the permanent prove- or a temporary prove-). Plants one deliberate bug at a time in the files the current change touched (or along a flow the user names), and for a migration only in the new system, never the old one it is compared with. A fresh agent then checks that change or flow with the skill and its answer keys, without knowing what was planted. Records whether it caught the bug, reverts exactly, and reports every miss with a suggested fix to the skill and to the flow's answer key. Use for /audit-prove-it, "can the verifier be fooled", "test the verify skill", "is our verification any good", "mutation test the verifier", or after creating or heavily editing a prove-* skill.
Audit Prove It
A verification skill that says PROVED on a broken app is worse than none: it gives false confidence. The only way to know it works is to break the app on purpose and see if it notices. This skill does that, safely, and reports every bug that slipped past.
Safety first
You are about to edit product code on purpose. These rules keep that from hurting anyone:
- Only in a git repo, only on local files. Never against a shared, staging or production instance.
- Back up before you plant. The files a change touched usually have uncommitted work in them, and
git checkoutwould throw it away. Before planting, copy the file byte for byte to.prove-it/audits/<UTC timestamp>/originals/<path>. Revert by copying it back. This is what makes the revert exact. - One planted bug at a time. Plant, check, revert, confirm the revert, then the next.
- Confirm every revert with
cmp <backup> <file>(exit 0) before moving on. If it doesn't match, stop everything and restore from the backup. Never usegit checkouton a file that had uncommitted changes. - Never commit, push or stash a planted bug. At the end,
git statusmust look exactly as it did at the start, plus any receipts in.prove-it/and the report.
1. Find the skill and baseline it
Find the prove-* skill (under .claude/skills/ or .agents/skills/, with the prove-it: marker comment). If there are none, stop and suggest /create-prove-it. The marker says which kind it is.
For a temporary migration skill, plant bugs in the new system only. The old system is the answer key. A bug planted there tests nothing, and it can leak into the snapshots the end step records. Its target is the migration's covered areas that have moved or are in progress.
Work out the target. By default it is the current change: the files touched by uncommitted work plus the commits on the branch since its base, found the way the skill's Scope section says. If the user names a flow ("audit checkout"), the target is that flow and the files it runs through. If the current change touches no product code, ask for a flow.
Record git status --porcelain so you can compare at the end.
Run the skill once on the untouched target. It must come back PROVED. If the clean change already fails, any "catch" later means nothing. Fix that first, or suggest /maintain-prove-it, and leave the failing surface out of the audit. Note the surfaces and flows the baseline receipt drove, and how it sorted each one (new or existing flow, and the answer key): that is the scope you hand the blind checker.
2. Plan the bugs
Plan 4 to 8 bugs in the target's files, spread across surfaces and layers, so the audit finds the blind spots instead of hitting the same check eight times. Prefer the lines the change itself added or edited: that is where a real bug in this change would be. references/mutation-catalogue.md lists bug types by layer. Aim them at the skill's "Could fool you" notes first, since those are its own admitted weak spots. Put at least one in shared code an existing flow runs through, to test that the skill counts that flow as touched, and, for a surface that copies another, one in the code both sides share, to test whether parity alone misses it.
Each bug must be:
- Real: something a plausible code change could cause, not a syntax error that crashes startup (that only tests the health check, and one of those is enough).
- User-visible: a user, or a program calling the app, would notice. If nobody could notice, no verifier should catch it.
- Small: a line or two, so the revert is trivial.
Write the plan as a table before you touch anything: id, flow, layer, file, the change, what a user would see.
3. Plant, check blind, revert
For each bug:
- Back up and plant. Copy the file to the originals folder, then make the change. Restart the app if the skill's Start section says code changes need a restart.
- Check blind. The checker must not know what you planted, or it will go looking for it, which proves nothing. If your agent can start a sub-agent (the Agent tool in Claude Code, or a subagent in Codex), give it only this: "Use the
<prove skill name>skill to prove these flows in this repo:<the flows and surfaces from the baseline receipt>, with these checks and answer keys:<each flow's check and answer key from the baseline receipt, including the spec it found>. Report PROVED or BROKEN with evidence." No hint about the bug, the file or the layer. The answer keys are not hints: a real checker gets them too. Hand it the flows by name instead of the diff, and tell it not to readgit diff,git status,git logor.prove-it/audits/: the planted bug sits in the working tree, so any of these would give it away. Tell it to keep any scratch files inside its run folder. If you can't start a sub-agent, run the skill's scripted checks only and mark the result "not blind" in the report. - Score it, and note what caught it: the skill's scripts, or the checking agent going off-script. A verifier whose scripts catch little relies on every futur