experiment-audit
Audit experiment integrity before claiming results. Uses cross-model review (external reviewer backend) to check for fake ground truth, score normalization fraud, phantom results, and insufficient scope. Use when user says \"审计实验\", \"check experiment integrity\", \"audit results\", \"实验诚实度\", or after experiments complete before writing claims.
wanshuiyin · 已有 AI 审核记录
安装与使用
运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。
npx skills add wanshuiyin/Auto-claude-code-research-in-sleep --skill experiment-audit技能说明
name: experiment-audit description: "Audit experiment integrity before claiming results. Uses cross-model review (external reviewer backend) to check for fake ground truth, score normalization fraud, phantom results, and insufficient scope. Use when user says "审计实验", "check experiment integrity", "audit results", "实验诚实度", or after experiments complete before writing claims." argument-hint: "[experiment-dir-or-results-path]" allowed-tools: Bash(*), Read, Write, Edit, Grep, Glob, mcp__codex__codex, mcp__codex__codex-reply, mcp__manual_review__review, mcp__manual_review__review_reply
Experiment Audit: Cross-Model Integrity Verification
🔒 Do not wrap this skill in
/loop,/schedule, orCronCreate. It is verdict-bearing — it judges experiment integrity. Re-running that verdict on a timer adds no new signal, and a loop that accepts its own output to decide when to stop crosses into self-acquittal (acceptance-gate.md). Schedule the external wait that precedes it — experiments done → then audit once. Seeshared-references/external-cadence.md.
Audit experiment integrity for: $ARGUMENTS
Why This Exists
LLM agents can produce fraudulent experimental results through:
- Fake ground truth — creating synthetic "reference" from model outputs, then reporting high agreement as performance
- Score normalization — dividing metrics by the model's own max to get 0.99+
- Phantom results — claiming numbers from files that don't exist or functions never called
- Insufficient scope — reporting 2-scene pilots as "comprehensive evaluation"
These are NOT intentional deception — they are failure modes of optimizing agents that lack integrity constraints. This skill adds that constraint.
Core Principle
The executor collects file paths. The external reviewer backend reads code and judges integrity. The executor does NOT participate in integrity judgment.
This follows shared-references/reviewer-independence.md and shared-references/experiment-integrity.md.
Constants
- REVIEWER_BACKEND =
codex— Default: Codex MCP (ultra). Override with— reviewer: oracle-profor Oracle MCP, or— reviewer: manualfor Manual Review MCP. If manual-review MCP is unavailable, stop and print the install command; do not fall back to Codex. Seeshared-references/reviewer-routing.md.
Reviewer Calling Convention
When calling the reviewer, branch on REVIEWER_BACKEND:
If REVIEWER_BACKEND = codex:
Use mcp__codex__codex for new review threads.
Use mcp__codex__codex-reply for follow-up rounds (reuse threadId).
If REVIEWER_BACKEND = manual:
Use mcp__manual_review__review for new review threads with:
prompt: [exact same prompt that would go to Codex]
config: {"model_reasoning_effort": "xhigh", "executor_model": "", "require_reviewer_model": true}
Save the returned threadId.
Use mcp__manual_review__review_reply for follow-up rounds with:
threadId: [saved manual-review threadId]
prompt: [follow-up prompt]
config: {"model_reasoning_effort": "xhigh", "executor_model": "", "require_reviewer_model": true}
Prompt fidelity: the manual prompt must be exactly the same text that Codex would receive. Review tracing applies equally to both backends.
Workflow
Step 1: Collect Artifacts (Executor — Claude)
Locate and list these files WITHOUT reading or summarizing their content:
Scan project directory for:
1. Evaluation scripts: *eval*.py, *metric*.py, *test*.py, *benchmark*.py
2. Result files: *.json, *.csv in results/, outputs/, logs/
3. Ground truth paths: look in eval scripts for data loading (dataset paths, GT references)
4. Experiment tracker: EXPERIMENT_TRACKER.md, EXPERIMENT_LOG.md
5. Paper claims: NARRATIVE_REPORT.md, paper/sections/*.tex, PAPER_PLAN.md
6. Config files: *.yaml, *.toml, *.json configs with metric definitions
A verdict-bearing manual response MUST begin with
Reviewer-Model: <exact-model-id> — pass the model THIS session is actually
running as in executor_model. Missing, unknown, or same-family identity
cannot acquit; emit REVIEW_UNAVAILABLE rather than guessing. If the executor
model cannot be named, manual review's cross-family claim is unprovable — say
so in the report instead of asserting it.
DO NOT summarize, interpret, or explain any file content. Only collect paths.
Step 2: Send to Reviewer
Based on the selected reviewer backend (see Reviewer Calling Convention), pass ONLY file paths and the audit checklist to the reviewer. The reviewer reads everything directly.
For codex, call mcp__codex__codex with:
model: gpt-5.6-solconfig: {"model_reasoning_effort": "ultra"}sandbox: read-onlycwd: [project directory]prompt: [the exact full prompt below]
For manual, call mcp__manual_review__review with:
config: {"model_reasoning_effort": "xhigh", "executor_model": "<actual executor model>", "require_reviewer_model": true}prompt: [the exact full prompt below]
Manual review cannot use Codex-only model, sandbox, or cwd; include the same file paths in the prompt so the user can inspect them.
Use this exact prompt for both backends:
You are an experiment integrity auditor. Start from the assumption that the
evaluation is compromised somewhere — your job is to find where. Be
adversarial. Trust nothing the author tells you — verify everything
yourself. Read ALL files listed below and check for the following fraud
patterns.
Files to read:
- Evaluation scripts: [list paths]
- Result files: [list paths]
- Experiment tracker: [list paths]
- Paper claims: [list paths]
- Config files: [list paths]
## Audit Checklist
### A. Ground Truth Provenance
For each evaluation script:
1. Where does "ground truth" / "refer
阅读完整来源 ↗