跳至正文
OpenAgentSkill
浏览已开放 · 目录记录日期 2026-11-21 · 自动更新和账户服务暂时关闭。
其他技能

autoresearch

Runs autonomous keep/discard experiments on a codebase to optimize a single metric for a fixed duration, in the style of karpathy/autoresearch. Use when the user says "autoresearch" (optionally with a focus, e.g. "autoresearch the optimizer"), asks to run experiments on a repo overnight, to hill-climb or optimize a metric autonomously, or points at a repo with a karpathy-style program.md.

av · 已有静态检查记录

安装与使用

运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。

npx skills add av/skills --skill autoresearch

技能说明


name: autoresearch description: Runs autonomous keep/discard experiments on a codebase to optimize a single metric for a fixed duration, in the style of karpathy/autoresearch. Use when the user says "autoresearch" (optionally with a focus, e.g. "autoresearch the optimizer"), asks to run experiments on a repo overnight, to hill-climb or optimize a metric autonomously, or points at a repo with a karpathy-style program.md.

Autoresearch

Metric-gated experimental research on a codebase for a fixed duration. Subagents propose and run experiments. The metric decides what survives. The clock decides when to stop. You decide neither.

Setup ──► clock check ──► dispatch ONE experiment ──► verify ──► gate ──► log ──┐
              ▲                                                                 │
              └──────────────────── time remains ───────────────────────────────┘
              └── deadline passed ──► confirm best ──► final summary

Role and Iron Laws

You are the orchestrator: manage the clock, dispatch subagents, verify results, gate outcomes, keep the ledger. All experimental work — designing changes, editing code, running training/benchmarks — happens inside subagents.

1. THE CLOCK DECIDES WHEN TO STOP. NOT YOU.
2. THE METRIC DECIDES WHAT SURVIVES. NOT YOU.
3. ONE EXPERIMENT IN FLIGHT AT A TIME. NEVER PARALLEL.
4. EVERY FACT YOU NEED LIVES IN THE LEDGER FILES, NOT IN YOUR HEAD.

Law 3 deliberately deviates from the parallel-subagent pattern of sibling skills: experiments mutate shared state (one working tree, one branch, one compute resource). Parallel dispatch corrupts the protocol. Dispatch each subagent blocking (foreground) and wait for its result.

Law 4 is what lets a run survive 100+ experiments: your context will be compacted or reset overnight. See Resume.

Degrees of freedom are split on purpose:

  • Hypothesis selection is free. Subagents choose what to try; you and they may be ambitious, radical, creative — anything inside the focus and scope.
  • The protocol is fixed. Verify, gate, and log exactly as written below. No judgment calls except the simplicity criterion.

No subagent tool in this harness? Run the dispatch template yourself, one experiment per iteration, and still run Verify and Gate as a separate step afterwards — never trust the number you just produced without re-extracting it. Re-read the ledger TSV before every hypothesis instead of remembering it.

Inputs

InputRequiredExample
Target repoyes~/code/autoresearch
Metric + direction + extractionyesval_bpb, lower is better, grep "^val_bpb:" <log> | awk '{print $2}'
Experiment commandyesuv run train.py
Time budget per runyes5 minutes wall clock
Mutable scopeyestrain.py only
Durationyes"overnight" = 8 hours
Frozen scoperecommendedprepare.py, evaluation code, dependencies
Noise floorrecommendedrun-to-run spread of the metric, e.g. 0.002; unknown → measured in setup
Cost toleranceoptionalpeak VRAM may not exceed baseline by >10% + its extraction command
Holdout commandoptionala second eval on data the loop never sees; run once, at the end
Byproducts to clean between runsoptionalcheckpoints/, __pycache__
Research focusoptional"attention variants", "the optimizer only"
Run tagautodate-based, e.g. sep23

The extraction command must print one number and nothing else; write it so it reads a log path (<log>), not a fixed filename.

Research focus — a free-form directive from the user's prompt: a component, an idea family, a constraint, or a hunch. It bounds hypothesis selection in every dispatch; it never changes the gate. Record it verbatim in the ledger.

Native mode — if the target repo contains a karpathy-style program.md, read it first and adopt its mechanics (metric, commands, scopes, budgets) verbatim. Two things always follow this skill instead: logs and the ledger live in /tmp (see Setup), and the orchestrator gates, not the subagent. A user-stated focus overrides its open-ended charter for choosing hypotheses.

Metric integrity — the code that computes and prints the metric must sit in the frozen scope. If the user's scopes leave it mutable, flag it and get the scopes corrected before starting: experiments that can touch the metric computation produce incomparable numbers. The same applies to the eval data: if the mutable scope can choose or filter what the metric is computed on, that selection is frozen too.

Oracle, not claim — a subagent's reported number is a claim; the extraction you run yourself is the oracle, and only the oracle feeds the gate. A model grading its own run (self-score, LLM judge in the mutable scope) is never the metric. No deterministic extraction exists → this skill does not apply; say so instead of climbing on vibes.

If required inputs are missing and no program.md supplies them, ask the user once, up front, for everything at once. After setup, never ask again.

Setup

Track this checklist:

- [ ] 1. Clock: date +%s, deadline computed, run count estimated
- [ ] 2. Branch: fresh autoresearch/<tag> from current HEAD, tree clean
- [ ] 3. Ledger TSV + progress file created in /tmp
- [ ] 4. Baseline recorded as best; extraction proven; noise floor set
- [ ] 5. Executor pin recorded
  1. Clock. Record start timestamp and deadline. Runs available ≈ duration ÷ (per-run budget + ~2 min overhead). Fewer than ~5 → tell the user now; the run will mostly measure noise.
  2. Branch. git -C <repo> checkout -b autoresearch/<tag> from current HEAD. Must be fresh — if the branch exists or the tree is dirty, stop and tell the user. Every git command in this run uses git -C <repo> — never rely on cwd. This skill never runs git reset --hard, git clean, or git stash (the stash stack is shared with e
阅读完整来源 ↗