autoresearch
Runs autonomous keep/discard experiments on a codebase to optimize a single metric for a fixed duration, in the style of karpathy/autoresearch. Use when the user says "autoresearch" (optionally with a focus, e.g. "autoresearch the optimizer"), asks to run experiments on a repo overnight, to hill-climb or optimize a metric autonomously, or points at a repo with a karpathy-style program.md.
av · 已有静态检查记录
安装与使用
运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。
npx skills add av/skills --skill autoresearch技能说明
name: autoresearch description: Runs autonomous keep/discard experiments on a codebase to optimize a single metric for a fixed duration, in the style of karpathy/autoresearch. Use when the user says "autoresearch" (optionally with a focus, e.g. "autoresearch the optimizer"), asks to run experiments on a repo overnight, to hill-climb or optimize a metric autonomously, or points at a repo with a karpathy-style program.md.
Autoresearch
Metric-gated experimental research on a codebase for a fixed duration. Subagents propose and run experiments. The metric decides what survives. The clock decides when to stop. You decide neither.
Setup ──► clock check ──► dispatch ONE experiment ──► verify ──► gate ──► log ──┐
▲ │
└──────────────────── time remains ───────────────────────────────┘
└── deadline passed ──► confirm best ──► final summary
Role and Iron Laws
You are the orchestrator: manage the clock, dispatch subagents, verify results, gate outcomes, keep the ledger. All experimental work — designing changes, editing code, running training/benchmarks — happens inside subagents.
1. THE CLOCK DECIDES WHEN TO STOP. NOT YOU.
2. THE METRIC DECIDES WHAT SURVIVES. NOT YOU.
3. ONE EXPERIMENT IN FLIGHT AT A TIME. NEVER PARALLEL.
4. EVERY FACT YOU NEED LIVES IN THE LEDGER FILES, NOT IN YOUR HEAD.
Law 3 deliberately deviates from the parallel-subagent pattern of sibling skills: experiments mutate shared state (one working tree, one branch, one compute resource). Parallel dispatch corrupts the protocol. Dispatch each subagent blocking (foreground) and wait for its result.
Law 4 is what lets a run survive 100+ experiments: your context will be compacted or reset overnight. See Resume.
Degrees of freedom are split on purpose:
- Hypothesis selection is free. Subagents choose what to try; you and they may be ambitious, radical, creative — anything inside the focus and scope.
- The protocol is fixed. Verify, gate, and log exactly as written below. No judgment calls except the simplicity criterion.
No subagent tool in this harness? Run the dispatch template yourself, one experiment per iteration, and still run Verify and Gate as a separate step afterwards — never trust the number you just produced without re-extracting it. Re-read the ledger TSV before every hypothesis instead of remembering it.
Inputs
| Input | Required | Example |
|---|---|---|
| Target repo | yes | ~/code/autoresearch |
| Metric + direction + extraction | yes | val_bpb, lower is better, grep "^val_bpb:" <log> | awk '{print $2}' |
| Experiment command | yes | uv run train.py |
| Time budget per run | yes | 5 minutes wall clock |
| Mutable scope | yes | train.py only |
| Duration | yes | "overnight" = 8 hours |
| Frozen scope | recommended | prepare.py, evaluation code, dependencies |
| Noise floor | recommended | run-to-run spread of the metric, e.g. 0.002; unknown → measured in setup |
| Cost tolerance | optional | peak VRAM may not exceed baseline by >10% + its extraction command |
| Holdout command | optional | a second eval on data the loop never sees; run once, at the end |
| Byproducts to clean between runs | optional | checkpoints/, __pycache__ |
| Research focus | optional | "attention variants", "the optimizer only" |
| Run tag | auto | date-based, e.g. sep23 |
The extraction command must print one number and nothing else; write it
so it reads a log path (<log>), not a fixed filename.
Research focus — a free-form directive from the user's prompt: a component, an idea family, a constraint, or a hunch. It bounds hypothesis selection in every dispatch; it never changes the gate. Record it verbatim in the ledger.
Native mode — if the target repo contains a karpathy-style program.md,
read it first and adopt its mechanics (metric, commands, scopes, budgets)
verbatim. Two things always follow this skill instead: logs and the ledger
live in /tmp (see Setup), and the orchestrator gates, not the subagent.
A user-stated focus overrides its open-ended charter for choosing hypotheses.
Metric integrity — the code that computes and prints the metric must sit in the frozen scope. If the user's scopes leave it mutable, flag it and get the scopes corrected before starting: experiments that can touch the metric computation produce incomparable numbers. The same applies to the eval data: if the mutable scope can choose or filter what the metric is computed on, that selection is frozen too.
Oracle, not claim — a subagent's reported number is a claim; the extraction you run yourself is the oracle, and only the oracle feeds the gate. A model grading its own run (self-score, LLM judge in the mutable scope) is never the metric. No deterministic extraction exists → this skill does not apply; say so instead of climbing on vibes.
If required inputs are missing and no program.md supplies them, ask the user
once, up front, for everything at once. After setup, never ask again.
Setup
Track this checklist:
- [ ] 1. Clock: date +%s, deadline computed, run count estimated
- [ ] 2. Branch: fresh autoresearch/<tag> from current HEAD, tree clean
- [ ] 3. Ledger TSV + progress file created in /tmp
- [ ] 4. Baseline recorded as best; extraction proven; noise floor set
- [ ] 5. Executor pin recorded
- Clock. Record start timestamp and deadline. Runs available ≈ duration ÷ (per-run budget + ~2 min overhead). Fewer than ~5 → tell the user now; the run will mostly measure noise.
- Branch.
git -C <repo> checkout -b autoresearch/<tag>from current HEAD. Must be fresh — if the branch exists or the tree is dirty, stop and tell the user. Every git command in this run usesgit -C <repo>— never rely on cwd. This skill never runsgit reset --hard,git clean, orgit stash(the stash stack is shared with e