red-team
Use when the user wants to adversarially stress-test a guardrail, classifier, prompt, or API they own or are authorized to test, to surface the distinct ways it fails. Generates adversarial inputs, runs them through the target and a ground-truth oracle, logs every target-vs-oracle disagreement as a failure de-duplicated by technique class, and loops until rounds stop surfacing new classes. Produces a catalogue of distinct, reproducible failures — the attacker half of a find→fix setup. Not for patching the target, and not for attacking systems the user does not own or have permission to test.
gaasher · AI review on file
Installation
Read the source and dependencies before running a command. Copying does not install or execute anything.
npx skills add gaasher/Agent-Loop-Skills --skill red-teamAbout this skill
name: red-team description: > Use when the user wants to adversarially stress-test a guardrail, classifier, prompt, or API they own or are authorized to test, to surface the distinct ways it fails. Generates adversarial inputs, runs them through the target and a ground-truth oracle, logs every target-vs-oracle disagreement as a failure de-duplicated by technique class, and loops until rounds stop surfacing new classes. Produces a catalogue of distinct, reproducible failures — the attacker half of a find→fix setup. Not for patching the target, and not for attacking systems the user does not own or have permission to test. compatibility: Requires Python 3.9+ metadata: version: "0.1.0"
Red Team
An adversarial loop-until-dry. The artifact is a target system; the feedback signal is the count of
distinct failure classes you can surface. Each round you craft adversarial inputs aimed at new
weaknesses and run them through the target and a ground-truth oracle via tools/harness.py, which
records every disagreement as a failure and de-dupes by the class (technique) you label each input
with. You loop until fresh rounds stop finding anything new. This is only the find half of a
find→fix setup: it catalogues failures and never patches the target (see Pairing).
When to use
Use to harden a guardrail, classifier, content filter, prompt, or API that the user owns or is explicitly authorized to test — when the goal is a catalogue of distinct, reproducible failures, each an objective target-vs-oracle disagreement. A failure is a bypass (target allows what the oracle would block) or an over-block (target blocks what the oracle would allow).
Default: drive the loop with a runnable oracle so the signal is objective. Escape hatch: if the user has no runnable oracle, the oracle is your judgment against a written policy — apply it consistently and record the intended verdict per input. Not for fixing the target, and not for testing systems outside the user's authorization.
Setup
Resolve bindings interactively. If loop.run.yaml exists, load it, confirm the values in one line,
and skip to the loop. Otherwise: on Claude Code (the AskUserQuestion tool is available) infer a
likely value per binding and recommend it; on other hosts ask each as a quoted prompt. Then write
loop.run.yaml (format: examples/run.example.yaml) and confirm before creating any other files.
| binding | meaning | default | how to infer |
|---|---|---|---|
<target_cmd> | system under test: reads one input on stdin, prints a verdict (BLOCK/ALLOW, a label, a score). Never edited. | — | the guardrail/classifier/API entrypoint the user names |
<oracle_cmd> | ground-truth verdict for the same input. A failure is target != oracle. | — | a reference checker / policy impl; else your judgment vs a written policy |
<candidates_file> | each round's candidates, JSONL {id, text, class}; class is the technique the harness de-dupes on | <sandbox_root>/candidates.jsonl | — |
<failures_log> | append-only log of confirmed failures | <sandbox_root>/failures.jsonl | — |
<sandbox_root> | where candidates, failures, and the ledger live | ./sandbox | — |
<budget> | max rounds | 8 | — |
<patience> | stop after N consecutive rounds with no new failure class | 2 | — |
The signal comes from tools/harness.py. Run it each round:
python3 <skill_dir>/tools/harness.py --target "<target_cmd>" --oracle "<oracle_cmd>" \
--inputs <candidates_file> --log <failures_log>
It runs both commands on every candidate and prints one JSON object:
{tested, failures_this_run, new_classes, total_classes, examples}.
The loop
Copy this checklist and tick items off each round:
- Round 0 — probe: read the target's intended contract, run a small mixed batch through the harness to confirm wiring, note any failures it already reveals.
- Pick a fresh angle — a failure mode not yet surfaced (see the attack toolkit below).
- Generate a batch of candidates for that angle, each labeled with a root-cause
class; write to<candidates_file>. - Run
tools/harness.py; readnew_classesand the example failures. - Record: note which new classes this round added (the harness already appended the failures and de-duped). Append one ledger row.
- Stop when
<patience>consecutive rounds add no new class, or at<budget>. Else steer the next round at an untried angle.
Attack toolkit (draw from these, and invent your own):
- Obfuscation — case changes, spacing/punctuation, leetspeak, unicode homoglyphs, encoding.
- Paraphrase / synonyms — say the forbidden thing a different way; expand abbreviations.
- Boundary & context — embed the payload in benign text; multi-step or indirect phrasing.
- Over-block probes — benign inputs containing a trigger substring, to find false positives.
Labeling classes. A class is the root-cause technique — the single fixable weakness — not one
label per payload. Capitalizing password, apikey, and ssn are all the same class
(case-bypass), because one fix closes all of them; do not split them into case-password,
case-apikey, … That inflates the count so the loop never goes dry. Aim for a handful of root-cause
classes (e.g. case-bypass, leetspeak, spacing, missing-synonym, overblock), each
demonstrated by several payloads. Use a fresh class only for a genuinely new root cause; reuse a
class to add more evidence for one already found.
On stop, report: the catalogue of distinct failure classes with one reproducible example each, the bypass/over-block split, and — since the goal is a more robust target — a short suggested fix per class.
Ledger
<sandbox_root>/ledger.tsv, tab-separated, never commas in the text. Header round angle tested new_classes total_classes:
round angle tested new_classes total_classes
0 probe mixed batch 6 cas
Read the complete source ↗