threat-model-backtest
Backtest phase 3.6 of threat-model production against historical findings before sign-off. USE WHEN a draft model must prove it can uniquely route real reports. Builds a stratified producer-side corpus across components and contract dimensions, clusters large corpora by sink and attack class, and routes each item blind to one §1.17 disposition. Reports MODEL-GAP, contradictory or ambiguous routing, coverage, qualified §1.15 candidates, and §1.18 questions. The corpus is not published. DO NOT USE FOR: drafting, sidecar generation, or triaging one new finding.
alpha-omega-security · 已有静态检查记录
安装与使用
运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。
npx skills add alpha-omega-security/threat-model --skill threat-model-backtest技能说明
name: threat-model-backtest description: >- Backtest phase 3.6 of threat-model production against historical findings before sign-off. USE WHEN a draft model must prove it can uniquely route real reports. Builds a stratified producer-side corpus across components and contract dimensions, clusters large corpora by sink and attack class, and routes each item blind to one §1.17 disposition. Reports MODEL-GAP, contradictory or ambiguous routing, coverage, qualified §1.15 candidates, and §1.18 questions. The corpus is not published. DO NOT USE FOR: drafting, sidecar generation, or triaging one new finding. argument-hint: ''
Threat Model — Backtest (validate against history)
Phase 3.6. A draft that has never been backtested is untested software. The project's own history is a free test suite; use it before presenting the draft for sign-off. This is a producer-side quality gate — the corpus does not go into the published document (per the leave-out list: CVE history is not the threat model).
Procedure
-
Assemble a stratified corpus — start with the last 10–30 inbound security findings: published advisories, reports closed as "not a bug" / "by design", issues labeled
security, and scanner/fuzzer/AI-analysis output. If a vendoredsecurity-context.mdis present in the working directory (a runner's pre-fetch of exactly this material), seed the corpus from it before searching elsewhere. Prefer contested items, but do not let recency or controversy leave component families or contract dimensions untested.- Cover every in-scope component family.
- Cover every applicable contract-dimension row: numeric limits, failure atomicity, topology, callbacks, serialization, lifecycle, concurrency, and resource complexity.
- For a large corpus, cluster by
(component, sink, attack class, required attacker capability)and route at least one representative from every cluster. Increase the corpus beyond 30 when necessary to avoid an untested cluster; report both item count and cluster count. - Record each item's actual historical outcome where one exists —
fixed,wontfix,by-design,out-of-scope, orunknown— with the advisory or issue URL it came from. That label is the ground truth step 3 scores against, and without it the backtest cannot fail. Set the outcome aside while routing (step 2 is blind); compare only afterwards. - When no historical record is reachable, say so rather than inventing
one. Synthesize cases to exercise the matrix, mark every one
synthesized, and write the §1.1 note verbatim: "no historical corpus was available; the backtest routed N synthesized cases only." A self-invented corpus reported as history is worse than no backtest, because it reads as evidence.
-
Route each item blind — using only the draft (not hindsight knowledge of how it was actually resolved), apply the §1.1 triager quick-start and assign exactly one §1.17 disposition, citing the licensing section. Routing rules and the closed disposition set are in output-structure.md.
-
Score the routing. The two directions of error are not symmetric. Wrongly closing a real vulnerability is far worse than wrongly escalating a non-finding: an over-escalating model wastes maintainer time, an over-closing one hands a reporter a "not a bug" on a live issue. Score accordingly — this asymmetry decides every fix below.
Signature Meaning Fix Closes an item the project actually fixed the model closes a true positive — the one disqualifying outcome Blocking. Fix before sign-off. Narrow the licensing §1.12 disclaimer, §1.7 trusted marking, or §1.3 scope line until the item routes VALIDor escalates. Never widen a disclaimer to reach a close.Routes to MODEL-GAPthe model is missing a decision propose, in this order: an unresolvedmatrix row plus a proposed-answer §1.18 question; a conditional §1.11 guarantee; or an explicit §1.12 disclaimer. Prefer the option that leaves the report escalating over the option that closes itRoutes plausibly to two or more dispositions two sections overlap or contradict sharpen them until the routing is unique Routes to a disposition that contradicts how the maintainer actually resolved it the model is wrong, or the historical call was high-value §1.18 question — do not paper over it Disclaiming is the cheapest way to make a
MODEL-GAPdisappear, which makes it the easiest way to pass this gate while making the model worse. A disclaimer added because a corpus item routed badly is reverse-engineered from the answer: it must still be true of the project as it is, cite a real source, and stay inside the scope that source covers. -
Feed §1.15 — a recurring pattern is a candidate known non-finding only when its final outcome closes as
BY-DESIGN: property-disclaimedor an already-establishedKNOWN-NON-FINDING. Never promoteVALID-HARDENINGorMODEL-GAPinto a non-finding, and never promote anOUT-OF-MODEL:*route: a pattern that closes because the code is out of scope, the build is unsupported, or the root cause sits in a dependency keeps that disposition.KNOWN-NON-FINDINGis first in the precedence order, so relabelling one of those routes promotes it above the very checks that decided it. Require documented/maintainer provenance and record the exact component or sink, symptom/attack class, preconditions, and stable claim or obligation IDs that discharge it; textual resemblance alone is not a match. Every candidate must satisfy the four §1.15 rules inoutput-structure.md— in particular, no entry may match on the reporter's evidence