ab-equivalence
Compare two versions of an LLM-directed document - an original (teacher) and a candidate (student) - across a transfer set and return a per-case behavioural-equivalence verdict plus an efficiency signal. A transform-agnostic library capability other skills compose to gate a transform on behavioural sameness. TRIGGER when asked to A/B test two versions of a prompt / instruction / skill, to check whether a rewritten or compressed document still behaves the same as the original, to validate behavioural equivalence between two document versions, to gate a transform on no-regression, or when a skill needs the run-the-runner-on-both-versions-and-judge-equivalence capability.
bjcoombs · 已有静态检查记录
安装与使用
运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。
npx skills add bjcoombs/ai-native-toolkit --skill ab-equivalence技能说明
name: ab-equivalence description: "Compare two versions of an LLM-directed document - an original (teacher) and a candidate (student) - across a transfer set and return a per-case behavioural-equivalence verdict plus an efficiency signal. A transform-agnostic library capability other skills compose to gate a transform on behavioural sameness. TRIGGER when asked to A/B test two versions of a prompt / instruction / skill, to check whether a rewritten or compressed document still behaves the same as the original, to validate behavioural equivalence between two document versions, to gate a transform on no-regression, or when a skill needs the run-the-runner-on-both-versions-and-judge-equivalence capability."
A/B equivalence - a transform-agnostic behavioural-equivalence capability
A thin capability that compares two versions of an LLM-directed document - an original (the teacher) and a candidate (the student) - across a transfer set, and returns a per-case verdict on whether the candidate still induces the behaviour the original induced.
It is transform-agnostic: it judges behavioural equivalence between two versions and neither knows nor cares which transform produced the candidate. It therefore serves every optimizer transform that claims to preserve behaviour - compression today, directive-clarity next - not just compression. It is a library capability other skills compose: semantic-compress invokes it to gate a distillation, and skill-forge exposes it alongside its own quality gate. It does not judge absolute quality ("is this skill good?") - that is a different question answered by different judges. A/B equivalence judges sameness between two versions ("does the candidate still do what the original did?").
This skill owns the runner (references/runner-prompt.md, the pure-wrapper template, paths relative to this skill directory). The runner is the shared execution primitive: it applies one version of a document to one case input and returns a transcript and self-report. Skills that need behavioural comparison compose this capability rather than re-implementing the runner.
The runner ships in two variants, both pure wrappers returning the same six self-report fields (see references/runner-prompt.md):
- the skill variant (default) - the document is invoked on demand against a case input;
- the instruction-file variant - the document is an always-loaded agent instruction file (
CLAUDE.md,AGENTS.md,GEMINI.md,.cursor/rules/*,.github/copilot-instructions.md); the runner is handed only that file as its operating context plus a realistic repo task, and runs read-only / sandboxed (it states the actions it would take, never mutating the repo).
The variant is the caller's choice (skill-forge's artifact-type detection selects it); both produce a transcript the equivalence judge and skill-forge's lenses read identically.
Input contract
| Input | Required | Notes |
|---|---|---|
original | yes | Path to the teacher document - the version whose behaviour is the equivalence target. |
candidate | yes | Path to the student document - the transformed version under test. |
transfer_set | yes | Array of cases spanning the test taxonomy (happy / edge / adversarial / composition). The transfer set is the operational definition of the behaviour being preserved, so its breadth bounds the safety of the conclusion. |
The caller (e.g. semantic-compress) owns deriving and confirming the transfer set; this capability consumes it. A thin transfer set yields a weak equivalence claim - the caller is responsible for flagging coverage, and the output records it.
Mechanism
For each case in the transfer set:
- Run the runner (
references/runner-prompt.md, the pure-wrapper prompt) once withoriginalas the skill draft, on the case input, producing the teacher transcript. - Run the same runner once with
candidateas the skill draft, on the identical case input, producing the candidate transcript. - Hand both transcripts to the equivalence judge (
references/equivalence-judge-prompt.md- a focused compare-two-transcripts judge), which emits the per-case verdict and efficiency signal.
The runner and runner-prompt are the only execution primitive; the equivalence judge is the one comparison component, distinct from any absolute-quality lens. The judge compares observed behaviour to observed behaviour, never the candidate against what the original document says it should do. The full contract and schema are in references/ab-equivalence.md; the judge prompt and decision rule are in references/equivalence-judge-prompt.md.
Baseline caching (the teacher is captured once)
The original never changes across a multi-round transform loop, so its transcript per case is captured once and reused across every round. Only the candidate is re-run each round. This is a hard rule, not an optimization: re-running the teacher each round wastes runner budget and risks introducing teacher-side noise that the judge would mistake for a candidate change. The caller passes the cached teacher transcripts back in on rounds >= 2; this capability re-runs only the candidate. A budget ceiling on candidate re-runs belongs to the caller's loop, not here.
Verdict categories
Per case, the judge returns exactly one verdict:
| Verdict | Meaning | What it must cite |
|---|---|---|
equivalent | The candidate induced every behaviour and discipline the original induced. Incidental wording differences with no behavioural consequence are still equivalent. | Nothing required beyond the verdict. |
candidate-regressed | A behaviour or discipline the original induced is absent in the candidate. This is the failing verdict. | The specific behaviour lost - the discipline, step, or output the original produced and the candidate did not. |
candidate-diverged |