Skip to content
OpenAgentSkill
Browsing is open · Catalog captured 2026-11-21 · Automatic updates and account services are paused.
Other skills

planted-truth

Synthetic-truth controls for analysis pipelines. Use before running any statistical fit, solver, audit, or search on real data — the pipeline must first prove it can recover a known planted answer and catch a deliberately corrupted input.

BootLoops-ai · Static checks on file

Installation

Read the source and dependencies before running a command. Copying does not install or execute anything.

npx skills add BootLoops-ai/skills --skill planted-truth

About this skill


name: planted-truth description: Synthetic-truth controls for analysis pipelines. Use before running any statistical fit, solver, audit, or search on real data — the pipeline must first prove it can recover a known planted answer and catch a deliberately corrupted input.

planted-truth — the pipeline runs on synthetic truth first

A pipeline that has only ever seen real data has never been tested, because nobody knew what the right answer was. The only inputs whose correct output is known are the ones you construct. Construct them, run them, and do it before real data is touched.

Real data cannot grade a pipeline. Whatever comes out looks like a finding: a slope, an evidence ratio, a list of anomalies, an empty list of anomalies. If the pipeline drops half its input on a parsing error, the output is smaller and still looks like a finding. If a sign convention is inverted, the conclusion reverses and still looks like a finding. The one situation in which output can be graded is when the answer was written down first — a plant: data generated from known parameters, fed through the full analysis, with recovery demanded to the accuracy the real analysis will claim.

The order rule is not a nicety. Controls designed after the real output has been inspected drift toward blessing it: the plant's parameters get chosen in the region where the pipeline is already known to behave, the corruption tests get chosen from the failure classes already ruled out, and the tolerance gets set just wide enough for what was seen. Plant first, look second. Once real output has been seen, the controls you design are no longer independent of it.

The procedure

0. Freeze the controls before the first real run. Write down the plants, the corruptions, the null tests, and the pass criteria for each while the real data is still unopened. A control invented later, to answer a doubt about a result already in hand, is evidence of much less.

1. Recover a plant, through the full path. Generate data from the model with known parameters and confirm the pipeline recovers them, to the accuracy the real analysis will claim. Two hard requirements hide in that sentence. First, the full path: the plant goes through the exact production entry point, same configuration, same options, same file formats — not a simplified call that skips the reader, the preprocessor, or the assembly step, because those are precisely where pipelines break. Second, the claimed accuracy: if the analysis will report an error bar, the planted value must come back inside it; if it will report evidence for a model, the plant generated under that model must yield that verdict. Recovery to worse accuracy than the claim tests a weaker claim than the one being made. Plant more than once, and include awkward corners of the parameter range along with the comfortable middle.

2. Catch corruptions. Feed the pipeline inputs deliberately broken in the ways it is supposed to detect — a sign flip, two swapped rows, two swapped column labels, a block scaled by a constant, a shifted grid, a duplicated record, a truncated file — one corruption at a time, and confirm every one is caught, loudly, at the step that claims to catch it. Run a clean twin alongside: an uncorrupted copy that must pass, proving the alarm is responding to the corruption and not to everything. A checker that has never fired is untested; a checker that fires on everything is noise.

3. Null the machinery. Where the pipeline sums, averages, or assembles contributions: push a table of zeros through the full path and demand exactly the baseline back — not approximately, exactly, because "approximately zero" is where sign errors and double-counting hide. Re-insert a component the system already contains and demand its recorded effect back, identically. Nulls test the plumbing separately from the statistics, and plumbing is where most real failures live.

4. Author the fixtures independently of the code under test. The expected answer must not come from the pipeline being tested. Generate the plant by construction — write the answer first, then produce data from it — or with a separate implementation. An expected-output file regenerated from the current code turns the test into a check that the code agrees with itself; every bug present at generation time is baked into the fixture and certified forever after.

5. Prove each control can fail. For every check in the battery, break its input once on purpose and watch it fire. This is the only way to find the checks that cannot fail — the aliased comparison, the unreachable assertion, the tolerance that spans the whole range. A control's first demonstrated failure is its birth certificate; before that it is a hope.

6. Record the controls with the result. The plants, the catches, and the nulls are part of the deliverable, not scaffolding to delete. A result whose controls were run but discarded cannot be distinguished, later, from a result whose controls were never run — and later is when the question gets asked.

7. When a control fires on real data: stop. Diagnose to the root before any further run. The forbidden move is the plausible benign story — "that check is oversensitive", "it's probably the known formatting quirk" — followed by an override. A fired control that gets explained away is worse than no control: it converts a working alarm into false confidence, and it trains everyone touching the pipeline to override the next one.

Failure modes the controls exist to catch

Each is a class seen in practice, in agent-built and human-built pipelines alike. The vignette states the mechanism.

  • The check that cannot fail. A comparison of a quantity against itself through an aliased variable; an assertion inside a branch nothing reaches; a tolerance wider than the range of possible answers. The test suite is green from the day it is written to the day the pipeline dies, and it was never once
Read the complete source ↗