blue-team

--- name: blue-team description: > Use when the user has concrete failing cases in code or a guardrail/classifier/filter/prompt/API they own — a red-team failure catalogue OR a CI/CD test-failure report (failing pytest/JUnit tests) — and wants the target patched until those failures are closed without breaking what already works. It points straight at the failed cases (normalize any source with tools/ingest.py), fixes one root-cause class per iteration, and re-checks with tools/verify.py — oracle mode against a red-team oracle, or tests mode against the test suite — keeping a patch only if it closes a class while nothing that passed before regresses, else reverting; loops until every class is closed (dry) or the budget runs out, then opens a pull request with the patch set. The defensive fixer half of a find→fix setup. Not for discovering new failures (that is red-team), and not for editing the oracle, tests, or holdout that define ground truth. compatibility: Requires Python 3.9+; git + the gh CLI for the pull-request handoff (degrades to a patch series). metadata: version: "0.1.0" ---

# Blue Team

A **defensive fixer** loop — the inverse of `red-team`. The artifact is the **target, now writable**; the feedback signal is two-part, like `optimize-loop`: a **gate that must hold** (nothing that passed before regresses) and a **metric that must drop** (the count of open failure classes, toward zero). You point it at a set of **concrete failed cases** and fix them one root-cause class at a time. Each iteration you patch one class, then run `tools/verify.py`, and keep the patch only if it closes the class with no regression, else revert. You loop until every class is closed (**dry**) or the budget runs out, then hand the patch set off as a pull request. This is the *fix* half of a find→fix setup (see [Pairing](#pairing)).

The failed cases come from a real source; `tools/ingest.py` normalizes any of them into one catalogue: - **`oracle` mode** — a `red-team` `failures.jsonl`: each case is an input where the target's verdict disagrees with a ground-truth **oracle**. A case is closed when target and oracle now agree; a regression is a benign `<holdout>` input that newly disagrees (most often a new over-block). - **`tests` mode** — a **CI/CD test-failure report** (`pytest --junitxml` / JUnit XML, or a list of failing node ids): each case is a failing test. A case is closed when its test now passes; a regression is any *other* test that was passing and now fails.

## When to use Use to fix a concrete set of failing cases in code or a guardrail/classifier/filter/prompt/API the user owns — a red-team catalogue, or the failing tests from a CI run — driving the open-class count to zero without breaking what worked. A `class` is the root-cause group the loop closes as a unit (a red-team technique, or a CI failure area / test class).

Default: pick the mode that matches the source (`oracle` for red-team, `tests` for CI/CD). Escape hatch: in `oracle` mode with no separate functional test suite, the `<holdout>` alone is the regression guard; in `tests` mode the suite's own previously-passing tests are the guard. Not for finding new failures (run `red-team`), and not for editing the ground truth (the oracle, the tests, or the holdout).

## Setup Resolve bindings interactively. If `loop.run.yaml` exists, load it, confirm the values in one line, and skip to the loop. Otherwise: on Claude Code (the `AskUserQuestion` tool is available) infer a likely value per binding and recommend it; on other hosts ask each as a quoted prompt. Then write `loop.run.yaml` and confirm before creating any other files. Two worked configs: `examples/run.example.yaml` (oracle mode) and `examples/tests.run.yaml` (tests mode).

| binding | meaning | default | how to infer | |---|---|---|---| | `<source>` | where the failed cases come from: `oracle` (red-team) or `tests` (CI/CD) | — | red-team `failures.jsonl` → `oracle`; failing pytest/JUnit → `tests` | | `<target_files>` | the file(s) the loop may edit to fix the target | — | the source/guardrail/classifier behind the failures | | `<catalogue>` | the failed cases to close, JSONL; build it with `tools/ingest.py` (see below) | `<sandbox_root>/catalogue.jsonl` | red-team's `<failures_log>`, or a JUnit report | | `<oracle_cmd>` | *(oracle mode)* ground-truth verdict (frozen), same stdin→verdict contract as red-team | — | a reference checker / policy impl | | `<holdout>` | *(oracle mode)* benign inputs that must keep passing (regression guard) | `<sandbox_root>/holdout.jsonl` | known-good inputs the oracle agrees on | | `<test_cmd>` | *(tests mode)* runs the suite and writes a JUnit XML; regressions read from it | — | `pytest --junitxml=<junit>` (or any runner that emits JUnit) | | `<junit>` | *(tests mode)* path to the JUnit XML `<test_cmd>` writes | `<sandbox_root>/junit.xml` | — | | `<iter_strategy>` | `branches` (one commit per kept fix → feeds the PR) or `snapshots` (folder per iter) | `branches` | dirty / non-git tree → snapshots | | `<pr_branch>` | branch the fixes land on and the PR opens from | `blue-team/<tag>` | today's date as `<tag>` | | `<sandbox_root>` | where snapshots + the ledger live | `./sandbox` | — | | `<budget>` | max iterations | 8 | — | | `<patience>` | give up on one class after N failed attempts → mark it a residual | 3 | — |

`<skill_dir>` is this skill's installed folder; substitute the real path when writing `loop.run.yaml`.

**Build the catalogue first** with `tools/ingest.py`, which normalizes any source into `{id, ..., class}`: ``` python3 <skill_dir>/tools/ingest.py --from red-team --in <failures.jsonl> --out <catalogue> # oracle mode python3 <skill_dir>/tools/ingest.py --from junit --in <report.xml> --out <catalogue> # tests mode ```

**The signal** each iteration is `tools/verify.py`, in the mode matching `<source>`: ``` # oracle mode — <target_cmd> runs <target_files>, e.g. "python3 ./guardrail.py" python3 <skill_dir>/tools/verify.py --target "<target_cmd>" --oracle "<oracle_cmd>" \ --catalogue <catalogue> --holdout <holdout> # tests mode — <test_cmd> writes the JUnit report verify.py then reads python3 <skill_dir>/tools/verify.py --test-cmd "<test_cmd>" --junit <junit> --catalogue <catalogue> ``` Either way it prints one JSON object: `{mode, open_classes, closed_classes, open_count, closed_count, regressions, regression_count, still_failing}`.

## The loop Copy this checklist and tick items off:

- [ ] Iteration 0 — baseline: run `tools/verify.py` in the `<source>` mode; record the **open classes** (should match the catalogue) as the current state and confirm `regression_count` is 0 — if it is not, the catalogue or holdout is dirty, so fix that before fixing the target. Log the baseline row. Save a pristine copy of `<target_files>` to `<sandbox_root>/iter0/` (snapshots mode) or note the branch base (branches mode) — this is the **baseline** the final handoff diffs against, and also the snapshot iteration 1 reverts to. - [ ] In `branches` mode, open the run on a fresh branch: `git checkout -b <pr_branch>`. - [ ] For iteration N (≥1): snapshot the **current, pre-patch** `<target_files>` to `iter<N>/` (or note the git HEAD) *before* editing, so a discard can restore exactly this state. - [ ] Pick **one** open class. Read its `still_failing` examples + the suggested fix from the catalogue, and patch `<target_files>` at the **root cause** — one fix should close *all* payloads of that class (e.g. normalize case once, not per-keyword). One class per iteration so each delta is attributable. - [ ] Check the signal: run `tools/verify.py`. **Discard** — restore the snapshot / `git reset --hard` — if `regression_count > 0` (the gate) or the targeted class is still open. `verify.py`'s regression check *is* the gate: in oracle mode a regression is a newly-broken `<holdout>` case (e.g. a fix that closes a bypass by over-blocking benign inputs); in tests mode it is any previously-passing test the patch broke. - [ ] **Keep** if there are no regressions and `open_count` strictly dropped. In `branches` mode commit it: `git commit -am "close <class>: <one-line fix>"`. Append a ledger row. - [ ] If a class resists `<patience>` attempts, mark it an **open residual** and move on rather than thrashing. Stop when `open_count` = 0 (dry), at `<budget>`, or when every remaining class is a residual. `<budget>` counts **attempts** (each keep *or* discard is one iteration), not classes closed — a discard still consumes the budget.

On stop, restore the working files to the **best** iteration (most classes closed, zero regressions) and report: classes closed vs residual, the failures resolved (oracle mode: the bypass/over-block split), and regressions avoided. Then open the pull request (see [Handoff](#handoff-the-pull-request)).

**Fix toolkit.** In `tests` mode the patches are ordinary bug-fixes, grouped by failure area and applied one area per iteration. In `oracle` mode (hardening a guardrail/filter), reach for these root-cause patterns, mirroring red-team's attack toolkit: - **Normalize before matching** — case-fold, de-leet (homoglyph/leet → letters), strip spacing and punctuation, NFKC-normalize unicode. One normalization step closes case / leetspeak / spacing classes. - **Broaden the policy** — add missing synonyms/expansions to the blocked set (the `missing-synonym` class), keyed to the oracle's categories, not ad-hoc strings. - **Tighten over-broad rules** — scope a match to whole words / the right context so benign inputs stop tripping it (the `overblock` class), the most common source of regressions.

Mind the **interaction order** (both modes): make the narrowing/over-broad fix *before* a sweeping one. A fix that strips separators (closing `spacing`) can re-collapse a benign input into an over-broad substring and silently reopen an `overblock` class — and likewise a broad code change can reopen a test a narrower fix had to protect. Fix the narrow/over-broad case first, then generalize.

## Ledger `<sandbox_root>/ledger.tsv`, tab-separated, never commas in the description. `regr` = `regression_count` this iteration (the gate: 0 is clean); `status` ∈ {`keep`,`discard`,`baseline`,`residual`}. Header `iter class_targeted regr open_classes status description`: ``` iter class_targeted regr open_classes status description 0 - 0 5 baseline catalogue: 5 open classes 1 case-bypass 0 4 keep case-fold the input before matching 2 leetspeak 1 4 discard de-leet regex also over-blocked a holdout input (regression) 3 leetspeak 0 3 keep de-leet via translate table, holdout clean 4 missing-synonym 0 2 keep add passphrase/credentials/api-key to the policy set 5 overblock 0 1 keep require whole-word "secret key", not bare "secret" 6 spacing 0 0 keep strip non-alphanumerics before matching — dry ``` Report the **best** iteration (open_classes lowest with `regr` 0), not necessarily the last.

## Constraints - **Only edit `<target_files>`.** The ground truth — the oracle + `<holdout>` (oracle mode) or the test suite (tests mode) — and `tools/verify.py` are frozen; editing what measures the fix manufactures a pass (same rule as red-team and optimize-loop). If the oracle or a test is itself wrong, that is a finding to report, not something to patch here. - **Fix the root cause, not the payload.** One fix should close every item of a class; patching a single example string while siblings still fail means the class is not closed. This mirrors red-team's class accounting, so the two loops agree on what "closed" means. - **The regression gate is non-negotiable.** A patch that breaks something that passed before — a new over-block/bypass (oracle mode) or a previously-passing test (tests mode) — is a regression, not progress; revert it regardless of how many classes it closes. Prefer a narrower fix over a sweepin