跳至正文
OpenAgentSkill
浏览已开放 · 目录记录日期 2026-11-21 · 自动更新和账户服务暂时关闭。
搜索与研究

dueling-autoresearch

Use when the user wants two approaches raced head-to-head on a single shared metric — e.g. a classical/algorithmic lane vs an ML/learned lane, or any two strategies for the same task. Each lane runs its own analysis-first research loop confined to its lane, the lanes share a scoreboard and may borrow ideas across the boundary without abandoning their identity, and a shared eval keeps the head-to-head honest; loops until interrupted, reporting the current leader. Not for improving a single approach in isolation (use a single-track research loop), and not for picking between two finished artifacts in one shot (that is a one-time comparison).

gaasher · 已有 AI 审核记录

安装与使用

运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。

npx skills add gaasher/Agent-Loop-Skills --skill dueling-autoresearch

技能说明


name: dueling-autoresearch description: > Use when the user wants two approaches raced head-to-head on a single shared metric — e.g. a classical/algorithmic lane vs an ML/learned lane, or any two strategies for the same task. Each lane runs its own analysis-first research loop confined to its lane, the lanes share a scoreboard and may borrow ideas across the boundary without abandoning their identity, and a shared eval keeps the head-to-head honest; loops until interrupted, reporting the current leader. Not for improving a single approach in isolation (use a single-track research loop), and not for picking between two finished artifacts in one shot (that is a one-time comparison). compatibility: Requires Python 3.9+ metadata: version: "0.1.0"

Dueling Autoresearch Loop

Two lanes work the same objective in parallel and race the same metric — by default a classical/algorithmic lane against an ML/learned lane (the lanes are user-named). Each lane runs its own analysis-first iteration via roles/TrackAgent.md, confined to its lane. Every round both lanes post to a shared duel_log.md scoreboard and may borrow ideas across the lane boundary — but each stays in its lane. The feedback signal is the shared <metric> on a shared eval: if the classical lane wins, that is a real result. Lanes support mixed code locations — a codebase lane edits existing repo files, a sandbox lane authors its own code — and an eval-parity gate keeps the scores comparable.

You are the orchestrator: each round you advance both lanes, update the scoreboard, and keep both honest. Do not pause for permission once the loop is running.

When to use

Use this to race two genuinely different approaches on one metric and keep them honest against the same eval — classical vs learned, two model families, two query strategies. Default to spawning both lanes in parallel and letting the scoreboard drive cross-lane idea borrowing; if a lane runs dry, push it to a more radical in-lane change or to borrow a fresh idea from the log. Not for tuning a single approach (use a single-track loop), and not for a one-shot comparison of two finished things.

The cast (all in this folder):

  • roles/TrackAgent.md — the per-lane researcher, instantiated once per lane.

Setup

Resolve bindings interactively. If loop.run.yaml exists in the working dir, load it, confirm the values in one line, and skip to the loop. Otherwise: on Claude Code (the AskUserQuestion tool is available) infer a likely value for each binding and present it as the recommended option; on other hosts ask each as a quoted plain-text prompt. Then write loop.run.yaml (format: examples/run.example.yaml) and confirm the values before creating any other files.

The host also decides spawn-or-degrade: on Claude Code spawn a real Agent per lane so the two run in parallel; otherwise adopt roles/TrackAgent.md inline and run the lanes sequentially.

Shared bindings (identical for both lanes — the honesty anchor):

bindingmeaningdefaulthow to infer
<metric>the single metric both lanes race; ground truth of the duel—ask; scan run logs for a printed score
<metric_direction>minimize or maximize—from the metric's nature (loss vs accuracy)
<gate>run budget unit: time or epochsepochsthe artifact's runner
<budget>epochs per run (or minutes if gate: time)5—
<sandbox_root>where snapshots, ledgers, and the duel log live./sandbox—
<iter_strategy>snapshots or branches (snapshots recommended — two lanes on one branch is simplest)snapshots—

Per-lane bindings (two lanes, default names classical and learned). Each lane has a code_location that decides which other fields it needs — never add code to the codebase:

fieldmeaningwhen
namelane namealways
code_locationcodebase or sandboxalways
run_cmdexisting entrypoint to run, e.g. python train.pyif codebase
editable_filesexisting repo files this lane may editif codebase
entrycommand run from inside <sandbox_root>/<lane>/iter<N>/, e.g. python run.pyif sandbox
  • codebase — the lane maps to existing code: edits its editable_files and runs run_cmd. Two codebase lanes must have non-overlapping editable_files.
  • sandbox — no implementation exists and none is added to the repo: the lane authors and runs its code inside <sandbox_root>/<lane>/iter<N>/ via entry.

Typical duel on a repo with one existing model: the learned lane is codebase (edits model.py/config.yaml, runs train.py); the classical lane is sandbox (authors its own code under <sandbox_root>/classical/iter<N>/). Nothing is added to the codebase, yet the classical lane is still built and iterated.

Eval-parity gate (the honesty anchor). Before starting, confirm both lanes report <metric> on the same held-out set, computed the same way, so the scores are comparable — state how each lane emits it (e.g. both print <metric>: to their run log). If they don't match, fix it first; the duel is meaningless otherwise. If gate: time, write a run_with_timeout.sh wrapper per lane (timeout $(( <budget> * 60 )) <entry-or-run_cmd> "$@").

Initialise the sandbox (after confirmation):

<sandbox_root>/
├── duel_log.md          ← shared channel + scoreboard (## Scoreboard, ## Round log; headers only)
├── <laneA>/results.tsv  ← lane A ledger, header only
└── <laneB>/results.tsv  ← lane B ledger, header only

Each lane's per-iteration work lives in <sandbox_root>/<lane>/iter<N>/ (analysis/, results/, the run log). A codebase lane's iter dir also holds code_snapshot/ (the pre-change copy for revert); a sandbox lane's iter dir holds the lane's actual code for that iterat

阅读完整来源 ↗