experiment-bridge

--- name: experiment-bridge description: "Workflow 1.5: Bridge between idea discovery and auto review. Reads EXPERIMENT_PLAN.md, implements experiment code, deploys to GPU, collects initial results. Use when user says \"实现实验\", \"implement experiments\", \"bridge\", \"从计划到跑实验\", \"deploy the plan\", or has an experiment plan ready to execute." argument-hint: "[experiment-plan-path-or-topic]" allowed-tools: Bash(*), Read, Write, Edit, Grep, Glob, Skill, mcp__codex__codex, mcp__codex__codex-reply ---

# Workflow 1.5: Experiment Bridge

Implement and deploy experiments from plan: **$ARGUMENTS**

## Overview

This skill bridges Workflow 1 (idea discovery + method refinement) and Workflow 2 (auto review loop). It takes the experiment plan and turns it into running experiments with initial results.

``` Workflow 1 output: This skill: Workflow 2 input: refine-logs/EXPERIMENT_PLAN.md → implement → GPT-5.6-Sol review → deploy → collect → initial results ready refine-logs/EXPERIMENT_TRACKER.md code (cross-model) /run-experiment for /auto-review-loop refine-logs/FINAL_PROPOSAL.md ```

## Constants

- **CODE_REVIEW = true** — GPT-5.6-Sol xhigh reviews experiment code before deployment. Catches logic bugs before wasting GPU hours. Set `false` to skip. - **AUTO_DEPLOY = true** — Automatically deploy experiments after implementation + review. Set `false` to manually inspect code before deploying. - **SANITY_FIRST = true** — Run the sanity-stage experiment first (smallest, fastest) before launching the rest. Catches setup bugs early. - **MAX_PARALLEL_RUNS = 4** — Maximum number of experiments to deploy in parallel (limited by available GPUs). - **BASE_REPO = false** — GitHub repo URL to use as base codebase. When set, clone the repo first and implement experiments on top of it. When `false` (default), write code from scratch or reuse existing project files. - **COMPACT = false** — When `true`, (1) read `idea-stage/IDEA_CANDIDATES.md` instead of full `idea-stage/IDEA_REPORT.md` if available, (2) append experiment results to `EXPERIMENT_LOG.md` after collection.

> Override: `/experiment-bridge "EXPERIMENT_PLAN.md" — compact: true, base repo: https://github.com/org/project`

## Inputs

This skill expects one or more of:

1. **`refine-logs/EXPERIMENT_PLAN.md`** (best) — claim-driven experiment roadmap from `/experiment-plan` 2. **`refine-logs/EXPERIMENT_TRACKER.md`** — run-by-run execution table 3. **`refine-logs/FINAL_PROPOSAL.md`** — method description for implementation context 4. **`idea-stage/IDEA_CANDIDATES.md`** — compact idea summary (preferred when `COMPACT: true`) *(fall back to `./IDEA_CANDIDATES.md` if not found)* 5. **`idea-stage/IDEA_REPORT.md`** — full brainstorm output *(fall back to `./IDEA_REPORT.md` if not found)*

If none exist, ask the user what experiments to implement.

## Workflow

### Phase 1: Parse the Experiment Plan

Read `EXPERIMENT_PLAN.md` and extract:

1. **Run order and milestones** — which experiments run first (sanity → baseline → main → ablation → polish) 2. **For each experiment block:** - Dataset / split / task - Compared systems and variants - Metrics to compute - Setup details (backbone, hyperparameters, seeds) - Success criterion - Priority (MUST-RUN vs NICE-TO-HAVE) 3. **Compute budget** — total estimated GPU-hours 4. **Method details** from `FINAL_PROPOSAL.md` — what exactly to implement

Present a brief summary:

``` 📋 Experiment plan loaded: - Milestones: [N] (sanity → baseline → main → ablation) - Must-run experiments: [N] - Nice-to-have: [N] - Estimated GPU-hours: [X]

Proceeding to implementation. ```

**Research-contract fallback**: if `idea-stage/docs/research_contract.md` does not exist yet (idea selected outside `/idea-discovery`, or an older run), create it now from `templates/RESEARCH_CONTRACT_TEMPLATE.md` using the selected idea + claims from the experiment plan. Downstream `/result-to-claim` and `/ablation-planner` read this file as the claims source, and session recovery (`docs/SESSION_RECOVERY_GUIDE.md`) depends on it existing.

### Phase 2: Implement Experiment Code

**If `BASE_REPO` is set** — clone the repo first: ```bash git clone <BASE_REPO> base_repo/ # Read the repo's README, understand its structure, find entry points # Implement experiments by modifying/extending this codebase ```

For each milestone (in order), write the experiment scripts:

1. **Check existing code** — scan the project (or cloned `base_repo/`) for existing experiment scripts, model code, data loaders. Reuse as much as possible.

2. **Implement missing pieces:** - Training scripts with proper argparse (all hyperparameters configurable) - Evaluation scripts computing the specified metrics - Data loading / preprocessing if needed - Baseline implementations if not already present - Fixed random seeds for reproducibility - Results saved to JSON/CSV for later analysis - Proper logging (wandb if configured in CLAUDE.md)

3. **Follow the plan's run order** — implement sanity-stage experiments first, then baselines, then main method, then ablations.

4. **Self-review before deploying:** - Are all hyperparameters from EXPERIMENT_PLAN.md reflected in argparse? - Is the random seed fixed and controllable? - Are results saved in a parseable format (JSON/CSV)? - Does the code match FINAL_PROPOSAL.md's method description?

### Phase 2.5: Cross-Model Code Review (when CODE_REVIEW = true)

**Skip this step if `CODE_REVIEW` is `false`.**

Before deploying, send the experiment code to GPT-5.6-Sol xhigh for review:

``` mcp__codex__codex: model: gpt-5.6-sol config: {"model_reasoning_effort": "xhigh"} prompt: | Review the following experiment implementation for correctness.

## Experiment Plan: [paste key sections from EXPERIMENT_PLAN.md]

## Method Description: [paste from FINAL_PROPOSAL.md]

## Implementation: [paste the experiment scripts]

Check for: 1. Does the code correctly implement the method described in the proposal? 2. Are all hyperparameters from the plan reflected in the code? 3. Are there any logic bugs (wrong loss function, incorrect data split, missing eval)? 4. Is the evaluation metric computed correctly? 5. **CRITICAL: Does evaluation use the dataset's actual ground truth labels — NOT another model's output as ground truth?** This is a common and severe bug. 6. Any potential issues (OOM risk, numerical instability, missing seeds)?

For each issue found, specify: CRITICAL / MAJOR / MINOR and the exact fix.

=== SCOPE LIMITS (these bound what you PROPOSE, never what you look for) === Report anything that is actually wrong here — including a rare-looking case, if this repo actually produces it. Then keep the fix in scope: 1. This is a RESEARCH-WORKFLOW tool, not a security paper. Verification is welcome; over-defense is not. Assume a cooperating operator on their own machine — a malicious local user is NOT in the threat model. 2. Do NOT propose SHA / hash / content-fingerprint / digest-binding schemes. Reporting a real defect in hashing code that already exists is fine. 3. NO speculative machinery: do not add feature flags, migration frameworks, compat layers, wrappers, pins, or similar mechanisms unless evidence shows a current repo defect they fix or an explicit existing invariant they must preserve. "Load-bearing", "compatibility", and "not scaffolding" are labels, not evidence. Point to the failing path/artifact or invariant, and check the proposal's factual premises, such as whether a named package version exists. 4. NO corner-case obsession: exotic encodings, symlink races, RTL text and millisecond races are out of scope unless you can show the case arises here. 5. Where a rubric or checklist is genuinely needed, do not over-mechanize judgement. A clear sentence a human reads beats a scored table nobody maintains. Exception: code that runs remote commands, starts a network service, or installs an MCP server runs on the user's machine with their credentials — trust-boundary findings there are in scope and the default is strict. Say plainly when something is correct. Do not manufacture findings. ```

**On review results:** - **No CRITICAL issues** → proceed to Phase 3 - **CRITICAL issues found** → fix them, then re-submit for review (max 2 rounds) - **Codex MCP unavailable** → skip silently, proceed to Phase 3 (graceful degradation)

### Phase 3: Sanity Check (if SANITY_FIRST = true)

Before deploying the full experiment suite, run the sanity-stage experiment:

``` /run-experiment [sanity experiment command] ```

Wait for completion. Verify: - Training loop runs without errors - Metrics are computed and saved correctly - GPU memory usage is within bounds - Output format matches expectations

If sanity fails → **auto-debug before giving up**. Budget: up to **2 patch attempts** on the same failure, then up to **2 clean reimplements** (4 total):

1. **Read the error** — parse traceback, stderr, and log files. (The same read-the-primary-artifact discipline applies to surprising REVIEWER verdicts: see `shared-references/review-tracing.md` § *Debugging With Traces*.) 2. **Diagnose** — classify the failure: - OOM → reduce batch size or enable gradient checkpointing - ImportError → install missing package - FileNotFoundError → fix path or download data - CUDA error → check GPU availability, reduce model size - NaN/divergence → reduce learning rate, check data preprocessing 3. **Fix and re-run** — apply the fix, re-run sanity 4. **Attempt 2+ still failing? → Call in Codex rescue** (if Codex plugin installed): Before the next retry, invoke `/codex:rescue` to get a second opinion on the root cause. Codex independently reads the code and error logs — it may spot issues Claude missed (wrong tensor shapes, subtle import shadowing, config mismatches, etc.). Apply its suggested fix, then re-run. - If `/codex:rescue` is not available (plugin not installed), continue with Claude's own diagnosis 5. **Both patch attempts failed on the same failure? → Discard and reimplement cleanly** (up to 2 reimplements). Rewriting the failing script from `EXPERIMENT_PLAN.md` / the research contract is a PEER move to another patch, not a last resort — a third patch on top of two wrong ones is usually worse than a clean rebuild. Delete ONLY the attempt's own code/scaffolding (scripts this phase generated); the plan, `EXPERIMENT_TRACKER.md`, user-authored project source, collected data, and results are never deletable (see `shared-references/external-cadence.md` § *Let a broken attempt restart, not just patch*). 6. **Budget exhausted (2 patches + 2 reimplements), or two reimplements failed the SAME way?** → stop, report the failure with all attempted fixes and error logs. Two clean reimplements failing identically usually means the plan or the environment is wrong — say so explicitly in the report, because that (not the broken build itself) is what needs the human. Do not proceed with broken code.

> Never give up on the first failure. Most experiment crashes are fixable without human intervention.

### Phase 4: Deploy Full Experiments

Deploy experiments following the plan's milestone order. **Route by job count**:

**Small batch (≤5 jobs per milestone)** → use `/run-experiment` directly: ``` /run-experiment [experiment commands] ```

**Large batch (≥10 jobs, multi-seed sweeps, or phase dependencies)** → use `/experiment-queue` for proper orchestration: ``` /experiment-queue [grid spec or manifest] ```

Auto-routing rule: if any milestone in `EXPERIMENT_PLAN.md` declares ≥10 jobs (e.g., `seeds: [42, 200, 201, ...]` × `N: [64, 128, 256]` × `n: [50K, 150K, 500K, 652K]` = 36 jobs) or declares teacher→student phase depende