跳至正文
OpenAgentSkill
浏览已开放 · 目录记录日期 2026-11-21 · 自动更新和账户服务暂时关闭。
AI 与知识库

ab-equivalence

Compare two versions of an LLM-directed document - an original (teacher) and a candidate (student) - across a transfer set and return a per-case behavioural-equivalence verdict plus an efficiency signal. A transform-agnostic library capability other skills compose to gate a transform on behavioural sameness. TRIGGER when asked to A/B test two versions of a prompt / instruction / skill, to check whether a rewritten or compressed document still behaves the same as the original, to validate behavioural equivalence between two document versions, to gate a transform on no-regression, or when a skill needs the run-the-runner-on-both-versions-and-judge-equivalence capability.

bjcoombs · 已有静态检查记录

安装与使用

运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。

npx skills add bjcoombs/ai-native-toolkit --skill ab-equivalence

技能说明


name: ab-equivalence description: "Compare two versions of an LLM-directed document - an original (teacher) and a candidate (student) - across a transfer set and return a per-case behavioural-equivalence verdict plus an efficiency signal. A transform-agnostic library capability other skills compose to gate a transform on behavioural sameness. TRIGGER when asked to A/B test two versions of a prompt / instruction / skill, to check whether a rewritten or compressed document still behaves the same as the original, to validate behavioural equivalence between two document versions, to gate a transform on no-regression, or when a skill needs the run-the-runner-on-both-versions-and-judge-equivalence capability."

A/B equivalence - a transform-agnostic behavioural-equivalence capability

A thin capability that compares two versions of an LLM-directed document - an original (the teacher) and a candidate (the student) - across a transfer set, and returns a per-case verdict on whether the candidate still induces the behaviour the original induced.

It is transform-agnostic: it judges behavioural equivalence between two versions and neither knows nor cares which transform produced the candidate. It therefore serves every optimizer transform that claims to preserve behaviour - compression today, directive-clarity next - not just compression. It is a library capability other skills compose: semantic-compress invokes it to gate a distillation, and skill-forge exposes it alongside its own quality gate. It does not judge absolute quality ("is this skill good?") - that is a different question answered by different judges. A/B equivalence judges sameness between two versions ("does the candidate still do what the original did?").

This skill owns the runner (references/runner-prompt.md, the pure-wrapper template, paths relative to this skill directory). The runner is the shared execution primitive: it applies one version of a document to one case input and returns a transcript and self-report. Skills that need behavioural comparison compose this capability rather than re-implementing the runner.

The runner ships in two variants, both pure wrappers returning the same six self-report fields (see references/runner-prompt.md):

  • the skill variant (default) - the document is invoked on demand against a case input;
  • the instruction-file variant - the document is an always-loaded agent instruction file (CLAUDE.md, AGENTS.md, GEMINI.md, .cursor/rules/*, .github/copilot-instructions.md); the runner is handed only that file as its operating context plus a realistic repo task, and runs read-only / sandboxed (it states the actions it would take, never mutating the repo).

The variant is the caller's choice (skill-forge's artifact-type detection selects it); both produce a transcript the equivalence judge and skill-forge's lenses read identically.

Input contract

InputRequiredNotes
originalyesPath to the teacher document - the version whose behaviour is the equivalence target.
candidateyesPath to the student document - the transformed version under test.
transfer_setyesArray of cases spanning the test taxonomy (happy / edge / adversarial / composition). The transfer set is the operational definition of the behaviour being preserved, so its breadth bounds the safety of the conclusion.

The caller (e.g. semantic-compress) owns deriving and confirming the transfer set; this capability consumes it. A thin transfer set yields a weak equivalence claim - the caller is responsible for flagging coverage, and the output records it.

Mechanism

For each case in the transfer set:

  1. Run the runner (references/runner-prompt.md, the pure-wrapper prompt) once with original as the skill draft, on the case input, producing the teacher transcript.
  2. Run the same runner once with candidate as the skill draft, on the identical case input, producing the candidate transcript.
  3. Hand both transcripts to the equivalence judge (references/equivalence-judge-prompt.md - a focused compare-two-transcripts judge), which emits the per-case verdict and efficiency signal.

The runner and runner-prompt are the only execution primitive; the equivalence judge is the one comparison component, distinct from any absolute-quality lens. The judge compares observed behaviour to observed behaviour, never the candidate against what the original document says it should do. The full contract and schema are in references/ab-equivalence.md; the judge prompt and decision rule are in references/equivalence-judge-prompt.md.

Baseline caching (the teacher is captured once)

The original never changes across a multi-round transform loop, so its transcript per case is captured once and reused across every round. Only the candidate is re-run each round. This is a hard rule, not an optimization: re-running the teacher each round wastes runner budget and risks introducing teacher-side noise that the judge would mistake for a candidate change. The caller passes the cached teacher transcripts back in on rounds >= 2; this capability re-runs only the candidate. A budget ceiling on candidate re-runs belongs to the caller's loop, not here.

Verdict categories

Per case, the judge returns exactly one verdict:

VerdictMeaningWhat it must cite
equivalentThe candidate induced every behaviour and discipline the original induced. Incidental wording differences with no behavioural consequence are still equivalent.Nothing required beyond the verdict.
candidate-regressedA behaviour or discipline the original induced is absent in the candidate. This is the failing verdict.The specific behaviour lost - the discipline, step, or output the original produced and the candidate did not.
candidate-diverged
阅读完整来源 ↗