Skip to content
OpenAgentSkill
Browsing is open · Catalog captured 2026-11-21 · Automatic updates and account services are paused.
Security

eval-leakage-audit

Audits whether a verification (eval/metric/experiment/holdout) actually secures independent external ground truth, or whether the designer, the model, and the scorer are just confirming each other in a circle — via an 18-pattern taxonomy. Read-only. Use before trusting any 'how we'll know it worked' — A/B tests, holdouts, scores, validation — especially when a result feels too clean or self-confirming. 한국어: '이 검증 순환논리 아닌지 봐줘', '이 평가 편파적이야?', '이 벤치마크 셀프체크야?'.

AlexZio00 · AI review on file

Installation

Read the source and dependencies before running a command. Copying does not install or execute anything.

npx skills add AlexZio00/sovereign-skills --skill eval-leakage-audit

About this skill


name: eval-leakage-audit skill_type: analysis tools: Read, Grep, Glob description: "Audits whether a verification (eval/metric/experiment/holdout) actually secures independent external ground truth, or whether the designer, the model, and the scorer are just confirming each other in a circle — via an 18-pattern taxonomy. Read-only. Use before trusting any 'how we'll know it worked' — A/B tests, holdouts, scores, validation — especially when a result feels too clean or self-confirming. 한국어: '이 검증 순환논리 아닌지 봐줘', '이 평가 편파적이야?', '이 벤치마크 셀프체크야?'." user_invocable: true concurrency_profile: read_only: true concurrency_safe: true destructive: none state_footprint: stateless not_for:

  • "Post-code-change checks (tests pass, diff scope, side effects) -> verification (different target: verification=the code change itself, this skill=the circularity of the measurement/eval design)" see_also:
  • agent: verification relation: "eval-leakage-audit=audits whether a measurement/benchmark/eval design is circular (target=the measurement tool), verification=post-completion checklist for code changes (target=the code)"

Eval Leakage Audit — Verification Circularity Audit

Purpose

When some verification (eval/metric/experiment/holdout) gives you confidence that something "worked," this skill checks whether that confidence actually comes from independent external evidence, or whether the people who designed the verification and the model that scores it are just confirming each other (circularity) — via 18 concrete patterns. A passing gate is not proof of quality — it only proves what it was designed to check, and this skill is the executable tool that actually tests that principle in practice.

Dominant variable: does this verification actually receive independent external ground truth, or are the designer, the model, and the scorer mistaking self-confirmation for a result?

Discard if: the target has no evaluation/verification/benchmark concept at all (pure code refactor, doc changes, etc.).

Trigger

  • "audit this eval for leakage", "check if this benchmark is circular", "is this evaluation biased?"
  • "이 검증 순환논리 아닌지 봐줘", "이 평가 편파적이야?", "이 벤치마크 셀프체크야?"
  • /eval-leakage-audit

Workflow

  1. Identify the target verification (eval/metric/experiment/holdout/"how will we know it worked"). Name its components — what plays the model role, what plays the scorer role, what plays the designer role, and which dataset is involved.
  2. Ask the core question: does independent external ground truth actually enter the loop?
  3. Apply all 18 patterns below to the target and report only the ones that actually fire (don't list patterns that didn't fire):
    1. Recall, not reason — the answer was replayed from something already known, not actually derived
    2. Wrong null hypothesis — the ablation only strips the surface label while the actual leaking signal stays in place
    3. Shared hallucination — two components confirm each other and dress up the circularity as a number
    4. Tautology — the scorer grades the bucket it drew itself (precise term: checker_overfit)
    5. Verifier = designer — an unreproducible, undisclosed recipe is passed off as a holdout
    6. Shared-pool bias — train/holdout come from the same labeler pool, so the same bias enters both sides
    7. Frame injection — the question itself hints at the answer
    8. Demand characteristics — the subject being measured knows it's being measured and behaves differently as a result
    9. Dual-fail-flag — when two independent subjects (models/implementations) fail on exactly the same hidden case, suspect a defect in the scorer itself before blaming the subjects. Independent failures coinciding by chance is unlikely — a match more likely points to a shared cause (a scorer bug, or an error in the hidden case itself)
    10. Asymmetric-baseline self-falsification — if the metric itself uses a biased, asymmetric baseline statistic, the null expectation isn't 50%, producing false positives. Before reporting a result, self-falsify the metric with a symmetric check
    11. Evidence-burn — a fixture a model arm has already observed is spent: don't reuse it afterward as independent evidence, a holdout, or a replication fixture. Generate a new variant for the next round, or retire the fixture
    12. Ungraded grader — trusting the scorer/answer-key/rubric itself without self-verifying it first. Before trusting it, run 4 gates: (1) reference-pass — a known-correct implementation scores full marks; (2) buggy-baseline-fail — a deliberately flawed implementation scores in the expected low-to-mid band (too low or too high signals a miscalibrated grader); (3) mutation-kill — the grader actually catches ≥3 plausible wrong answers; (4) dual-fail-flag — if both arms of a comparison fail on the exact same case, treat it as a grader/fixture defect signal, not a candidate failure
    13. Ceiling task — misreading a benchmark saturated at full marks for every candidate (zero discriminative power) as "no difference." If a k=1 pilot shows both arms scoring ≥95%, flag ceiling_detected — adding more fixtures of the same kind won't recover discriminative power. Respond by switching task families, promoting process-layer metrics (cost, verification cadence), or adding 1-2 decisive branching probes. Post-hoc discriminative-power indicators: ceiling_rate, score_sd, bucket_entropy, winner_flip_rate
    14. Respawn masking — in systems with respawn/reset logic (games, simulations, state machines), scoring by an instant state snapshot lets a respawn disguise failure as a pass (for example: a character dies, auto-respawns, and the snapshot moment shows only alive — the death vanishes). Score by session-wide deterministic invariants instead (for example: death/reset event count = 0, no cumulative resource loss) rather than a state snapshot. [borrowed from fabulous HC3-13]
    15. Pseudo-replication — cou
Read the complete source ↗