跳至正文
OpenAgentSkill
浏览已开放 · 目录记录日期 2026-11-21 · 自动更新和账户服务暂时关闭。
搜索与研究

karpathy

Use when the user wants the LLM to do its own ML research: a fully-autonomous loop that hacks the training code, runs it, and keeps changes that lower a single scalar metric (e.g. val_bpb). One agent proposes one change at a time, runs training in the user's env, keeps it only if the metric improves (advancing a git branch) else reverts, and loops forever until the human interrupts. A faithful adaptation of Karpathy's autoresearch. Not for the analysis-first variant that profiles before editing (that is ml-autoresearch), and not for a budgeted, plateau-stopping refactor.

gaasher · 已有 AI 审核记录

安装与使用

运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。

npx skills add gaasher/Agent-Loop-Skills --skill karpathy

技能说明


name: karpathy description: > Use when the user wants the LLM to do its own ML research: a fully-autonomous loop that hacks the training code, runs it, and keeps changes that lower a single scalar metric (e.g. val_bpb). One agent proposes one change at a time, runs training in the user's env, keeps it only if the metric improves (advancing a git branch) else reverts, and loops forever until the human interrupts. A faithful adaptation of Karpathy's autoresearch. Not for the analysis-first variant that profiles before editing (that is ml-autoresearch), and not for a budgeted, plateau-stopping refactor. compatibility: Requires Python 3.9+ metadata: version: "0.1.0"

Karpathy Autoresearch

This is an experiment to have the LLM do its own research. You are a completely autonomous researcher: you hack the training code with an idea, run it, keep the change if the metric improves and revert it if it doesn't, advancing a branch as you go — and you repeat forever, until the human interrupts you. The artifact is the <editable_files>; the feedback signal is one scalar <metric> (lower is better, e.g. val_bpb) read from the run. Training runs in the user's own environment via <run_cmd> — this skill installs nothing and imports nothing; it edits code, shells out, and reads the metric from the log.

When to use

Use this to leave an agent running on a single training script, optimizing one scalar metric hands-off, where any improvement is kept and the loop never stops on its own. Default to broad freedom inside <editable_files>; the only hard limit is that the run finishes within the budget without crashing. Not for the analysis-first variant that reasons about the data before each edit (that is ml-autoresearch).

Setup

Resolve bindings interactively (load loop.run.yaml and skip if it already exists; else, on Claude Code infer + recommend each via AskUserQuestion, otherwise ask as quoted prompts; write loop.run.yaml). Then work with the user to set up a fresh run:

  1. Choose <iter_strategy> — branches (one git commit per run; the original) or snapshots (one folder per run under <sandbox_root>/). Snapshots are safer on a dirty or gitignored tree; branches mirror Karpathy. Either is fully supported throughout the loop.
  2. Open the run — branches: agree on a run tag from today's date (e.g. mar5) and create the branch git checkout -b autoresearch/<tag> (it must not already exist; this is a fresh run). snapshots: no branch — each iteration gets its own <sandbox_root>/iter<N>/.
  3. Read the in-scope files — the repo is small; read them for full context: the README, the read-only harness that defines the metric (the <metric> ground truth — do not modify), and the <editable_files> you will hack (model/optimizer/training loop).
  4. Verify the env/data exists — confirm <run_cmd> can run (data shards, tokenizer, deps present). If not, tell the human the one command to prepare it (e.g. uv run prepare.py).
  5. Initialize results.tsv — create it with just the header row; the baseline is recorded after the first run. Leave it untracked (never commit it).
  6. Confirm and go — confirm the setup looks right, then kick off the experimentation.
bindingmeaningdefaulthow to infer
<metric>scalar to minimize; must be printed by the run (e.g. val_bpb)—grep the in-scope files / README for val_bpb, val_loss, error…
<run_cmd> / <entrypoint>the command that launches one training run—e.g. uv run train.py; from pyproject.toml/.venv/README
<editable_files>the file(s) you may hack — everything else is read-only—the training script(s); exclude data, the eval harness, configs you must not touch
<sandbox_root>where results.tsv (+ snapshots) live./sandbox—
<iter_strategy>branches (a git commit per run) or snapshots (a folder per run)snapshotssnapshots is safer off a dirty/gitignored tree; branches mirrors Karpathy
<gate> / <budget>time (wall-clock) or epochs, and its value—the run's existing time/epoch setting

The experiment loop

Each experiment is one training run on a fixed budget (<gate>/<budget> — wall-clock time or a fixed epoch count, excluding startup/compile). You launch it simply: <run_cmd> (e.g. uv run train.py). Because the budget is fixed you don't need to worry about training time — every run gets the same budget.

What you CAN do: Modify <editable_files> — this is the only file you edit. Everything is fair game: model architecture, optimizer, hyperparameters, training loop, batch size, model size, etc.

What you CANNOT do: modify the read-only harness or the evaluation (the <metric> is the ground truth); install new packages or add dependencies (use only what's already available).

The goal is simple: get the lowest <metric>. Since the budget is fixed, you don't need to worry about training time — it's always the budget. Everything is fair game: change the architecture, the optimizer, the hyperparameters, the batch size, the model size. The only constraint is that the code runs without crashing and finishes within the budget.

VRAM is a soft constraint. Some increase is acceptable for meaningful <metric> gains, but it should not blow up dramatically.

Simplicity criterion: All else being equal, simpler is better. A small improvement that adds ugly complexity is not worth it. Conversely, removing something and getting equal or better results is a great outcome — that's a simplification win. When evaluating whether to keep a change, weigh the complexity cost against the improvement magnitude. A 0.001 <metric> improvement that adds 20 lines of hacky code? Probably not worth it. A 0.001 <metric> improvement from deleting code? Definitely keep. An improvement of ~0 but much simpler code? Keep.

**Th

阅读完整来源 ↗