karpathy
Use when the user wants the LLM to do its own ML research: a fully-autonomous loop that hacks the training code, runs it, and keeps changes that lower a single scalar metric (e.g. val_bpb). One agent proposes one change at a time, runs training in the user's env, keeps it only if the metric improves (advancing a git branch) else reverts, and loops forever until the human interrupts. A faithful adaptation of Karpathy's autoresearch. Not for the analysis-first variant that profiles before editing (that is ml-autoresearch), and not for a budgeted, plateau-stopping refactor.
gaasher · 已有 AI 审核记录
安装与使用
运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。
npx skills add gaasher/Agent-Loop-Skills --skill karpathy技能说明
name: karpathy description: > Use when the user wants the LLM to do its own ML research: a fully-autonomous loop that hacks the training code, runs it, and keeps changes that lower a single scalar metric (e.g. val_bpb). One agent proposes one change at a time, runs training in the user's env, keeps it only if the metric improves (advancing a git branch) else reverts, and loops forever until the human interrupts. A faithful adaptation of Karpathy's autoresearch. Not for the analysis-first variant that profiles before editing (that is ml-autoresearch), and not for a budgeted, plateau-stopping refactor. compatibility: Requires Python 3.9+ metadata: version: "0.1.0"
Karpathy Autoresearch
This is an experiment to have the LLM do its own research. You are a completely autonomous researcher:
you hack the training code with an idea, run it, keep the change if the metric improves and revert it if
it doesn't, advancing a branch as you go — and you repeat forever, until the human interrupts you.
The artifact is the <editable_files>; the feedback signal is one scalar <metric> (lower is better,
e.g. val_bpb) read from the run. Training runs in the user's own environment via <run_cmd> — this
skill installs nothing and imports nothing; it edits code, shells out, and reads the metric from the log.
When to use
Use this to leave an agent running on a single training script, optimizing one scalar metric hands-off,
where any improvement is kept and the loop never stops on its own. Default to broad freedom inside
<editable_files>; the only hard limit is that the run finishes within the budget without crashing. Not
for the analysis-first variant that reasons about the data before each edit (that is ml-autoresearch).
Setup
Resolve bindings interactively (load loop.run.yaml and skip if it already exists; else, on Claude Code
infer + recommend each via AskUserQuestion, otherwise ask as quoted prompts; write loop.run.yaml).
Then work with the user to set up a fresh run:
- Choose
<iter_strategy>— branches (one git commit per run; the original) or snapshots (one folder per run under<sandbox_root>/). Snapshots are safer on a dirty or gitignored tree; branches mirror Karpathy. Either is fully supported throughout the loop. - Open the run — branches: agree on a run tag from today's date (e.g.
mar5) and create the branchgit checkout -b autoresearch/<tag>(it must not already exist; this is a fresh run). snapshots: no branch — each iteration gets its own<sandbox_root>/iter<N>/. - Read the in-scope files — the repo is small; read them for full context: the README, the
read-only harness that defines the metric (the
<metric>ground truth — do not modify), and the<editable_files>you will hack (model/optimizer/training loop). - Verify the env/data exists — confirm
<run_cmd>can run (data shards, tokenizer, deps present). If not, tell the human the one command to prepare it (e.g.uv run prepare.py). - Initialize
results.tsv— create it with just the header row; the baseline is recorded after the first run. Leave it untracked (never commit it). - Confirm and go — confirm the setup looks right, then kick off the experimentation.
| binding | meaning | default | how to infer |
|---|---|---|---|
<metric> | scalar to minimize; must be printed by the run (e.g. val_bpb) | — | grep the in-scope files / README for val_bpb, val_loss, error… |
<run_cmd> / <entrypoint> | the command that launches one training run | — | e.g. uv run train.py; from pyproject.toml/.venv/README |
<editable_files> | the file(s) you may hack — everything else is read-only | — | the training script(s); exclude data, the eval harness, configs you must not touch |
<sandbox_root> | where results.tsv (+ snapshots) live | ./sandbox | — |
<iter_strategy> | branches (a git commit per run) or snapshots (a folder per run) | snapshots | snapshots is safer off a dirty/gitignored tree; branches mirrors Karpathy |
<gate> / <budget> | time (wall-clock) or epochs, and its value | — | the run's existing time/epoch setting |
The experiment loop
Each experiment is one training run on a fixed budget (<gate>/<budget> — wall-clock time or a
fixed epoch count, excluding startup/compile). You launch it simply: <run_cmd> (e.g. uv run train.py). Because the budget is fixed you don't need to worry about training time — every run gets the
same budget.
What you CAN do: Modify <editable_files> — this is the only file you edit. Everything is fair
game: model architecture, optimizer, hyperparameters, training loop, batch size, model size, etc.
What you CANNOT do: modify the read-only harness or the evaluation (the <metric> is the ground
truth); install new packages or add dependencies (use only what's already available).
The goal is simple: get the lowest <metric>. Since the budget is fixed, you don't need to worry about
training time — it's always the budget. Everything is fair game: change the architecture, the optimizer,
the hyperparameters, the batch size, the model size. The only constraint is that the code runs without
crashing and finishes within the budget.
VRAM is a soft constraint. Some increase is acceptable for meaningful <metric> gains, but it should not
blow up dramatically.
Simplicity criterion: All else being equal, simpler is better. A small improvement that adds ugly
complexity is not worth it. Conversely, removing something and getting equal or better results is a great
outcome — that's a simplification win. When evaluating whether to keep a change, weigh the complexity
cost against the improvement magnitude. A 0.001 <metric> improvement that adds 20 lines of hacky code?
Probably not worth it. A 0.001 <metric> improvement from deleting code? Definitely keep. An improvement
of ~0 but much simpler code? Keep.
**Th
阅读完整来源 ↗