Skip to content
OpenAgentSkill
Browsing is open · Catalog captured 2026-11-21 · Automatic updates and account services are paused.
Search & research

experiment-pipeline

Guides structured 4-stage experiment execution with attempt budgets and gate conditions: Stage 1 initial implementation (reproduce baseline), Stage 2 hyperparameter tuning, Stage 3 proposed method validation, Stage 4 ablation study. Integrates with evo-memory (load prior strategies, trigger IVE/ESE) and experiment-craft (5-step diagnostic on failure). Use when: user has a planned experiment, needs to reproduce baselines, organize experiment workflow, or systematically validate a method. Do NOT use for debugging a specific experiment failure (use experiment-craft) or designing which experiments to run (use paper-planning).

CamusGIT · AI review on file

Installation

Read the source and dependencies before running a command. Copying does not install or execute anything.

npx skills add CamusGIT/EvoQuant --skill experiment-pipeline

About this skill


name: experiment-pipeline description: "Guides structured 4-stage experiment execution with attempt budgets and gate conditions: Stage 1 initial implementation (reproduce baseline), Stage 2 hyperparameter tuning, Stage 3 proposed method validation, Stage 4 ablation study. Integrates with evo-memory (load prior strategies, trigger IVE/ESE) and experiment-craft (5-step diagnostic on failure). Use when: user has a planned experiment, needs to reproduce baselines, organize experiment workflow, or systematically validate a method. Do NOT use for debugging a specific experiment failure (use experiment-craft) or designing which experiments to run (use paper-planning)." allowed-tools: "write_file edit_file read_file think_tool execute" metadata: author: EvoQuant version: '1.0.0' tags: [core, experimentation, experiment-design]

Experiment Pipeline

A structured 4-stage framework for executing research experiments from initial implementation through ablation study, with attempt budgets and gate conditions that prevent wasted effort. This follows the Experiment Tree Search design from the EvoQuant paper, where the engineer agent iteratively generates executable code, runs experiments, and records structured execution results at each stage.

When to Use This Skill

  • User has a planned experiment and needs to organize the execution workflow
  • User wants to systematically validate a novel method against baselines
  • User asks about experiment stages, attempt budgets, or when to move on
  • User needs to reproduce baseline results before testing their method
  • User mentions "experiment pipeline", "baseline first", "ablation study", "stage budget", "experiment execution"

The Pipeline Mindset

Experiments fail for two reasons: wrong order and no stopping criteria. Most researchers jump straight to testing their novel method without verifying their baseline setup, then wonder why results don't make sense. Others spend weeks tuning hyperparameters without a budget, hoping the next run will work.

The 4-stage pipeline solves both problems. It enforces a strict order (each stage validates assumptions the next stage depends on) and assigns attempt budgets (forcing systematic thinking over brute-force iteration).

Before Starting: Load Prior Knowledge

If coming from research-ideation, your research proposal (Step 7) provides the experiment plan — datasets, baselines, metrics, and ablation design — that maps directly to Stages 1-4 below.

Dataset rule (applies to every stage): the proposal's dataset plan is already scoped to what is actually available, not the paper's dataset. The local dataset usually differs from the paper's (universe, date window, vendor). So reproduction is a faithful reconstruction on the scoped local data, not an exact match to the paper's numbers. Compare against a tolerance justified by the documented dataset differences; reserve "within 2% of reported" for the rare case you run on the paper's own data.

Before entering the pipeline, load Experimentation Memory (M_E) from prior cycles:

  1. Refer to the evo-memory skill → Read M_E at /memory/experiment-memory.md
  2. Select the top-1 entry (k_E=1) most relevant to the current experiment domain by comparing each entry's Context and Category against the current problem
  3. The selected strategy informs hyperparameter ranges (Stage 2), debugging approaches (Stages 1-3), and training configurations across all stages
  4. If M_E doesn't exist yet (first cycle), skip this step and proceed — your results will seed M_E via ESE after pipeline completion

4-Stage Pipeline Overview

Each stage follows a generate → execute → record → diagnose → revise loop:

StageGoalBudget (N_E^s)Gate Condition
1. Initial ImplementationGet baseline code running and reproduce known results≤20 attemptsResult consistent with paper at the level the (local) data allows — tolerance justified by dataset differences, or within 2% only if running on the paper's own data
2. Hyperparameter TuningOptimize config for your setup≤12 attemptsStable config, variance < 5% across 3 runs
3. Proposed MethodImplement & validate novel method≤12 attemptsOutperforms tuned baseline on primary metric, consistent across 3 runs
4. Ablation StudyProve each component's contribution≤18 attemptsAll claims evidenced with controlled experiments

Each stage saves artifacts to /experiments/<project>/stageN_name/.

Project directory

Each research cycle writes to its own subdirectory under /experiments/, so multiple cycles in the same workdir never collide. At the start of a cycle, create the project directory once and use it for every stage:

  • Name: <id>_<date>_<slug> where
    • <id> — a short two-digit cycle index within this workdir (01, 02, …), so cycles sort chronologically and stay readable. Pick the next free index.
    • <date> — cycle start date, YYYYMMDD.
    • <slug> — kebab-case slug of the selected direction / proposal title (from research-ideation's /direction-summary.md "Selected for proposal extension" name, or the proposal title). Lowercase, ASCII, hyphen-separated, truncated to ~40 chars.
  • Example: /experiments/02_20260729_str-robust-turnover/.
  • All stage outputs, artifacts, and trajectory logs for this cycle go under this directory. The directory name is the cycle's identity in the workdir.
  • Re-runs / continuations of the same cycle reuse the same directory; do not create a new one per stage or per attempt.

The Stage Loop

Within every stage, repeat this cycle for each attempt:

  1. Generate: Form a hypothesis or plan for this attempt. What specifically will you try? What do you expect to happen?
  2. Execute: Run the experiment. Record exact configuration, code changes, and runtime.
  3. Record: Log results immediately using the stage log template. Include both me
Read the complete source ↗