ml-autoresearch
Use when the user wants an autonomous ML research loop that does more than blindly try changes. After every training run the agent analyses what actually happened inside the model — gradients, activations, embeddings, errors, data — and grounds the next change in that evidence. A `<literature>` on/off dial adds scientific-literature grounding: off behaves as a pure analysis-first loop; on searches papers, grades the evidence, and implements only what prior work supports. One change per run; loops forever until interrupted. Not for one-off training runs or hyperparameter sweeps.
gaasher · 已有 AI 审核记录
安装与使用
运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。
npx skills add gaasher/Agent-Loop-Skills --skill ml-autoresearch技能说明
name: ml-autoresearch
description: >
Use when the user wants an autonomous ML research loop that does more than blindly try changes.
After every training run the agent analyses what actually happened inside the model — gradients,
activations, embeddings, errors, data — and grounds the next change in that evidence. A <literature>
on/off dial adds scientific-literature grounding: off behaves as a pure analysis-first loop; on
searches papers, grades the evidence, and implements only what prior work supports. One change per
run; loops forever until interrupted. Not for one-off training runs or hyperparameter sweeps.
compatibility: Requires Python 3.9+
metadata:
version: "0.1.0"
ML Autoresearch Loop
This loop is analysis-first: every experiment is followed by a diagnostic pass that examines what
happened inside the model, and the next change is a hypothesis grounded in that evidence — not a guess.
The feedback signal is <metric> read from the run log; the analysis is the spine that decides what to
change. A <literature> dial (on/off) optionally grounds each change in prior work via the sibling
literature-search skill. One change per iteration, so each metric move is attributable.
You are the researcher. Do not pause to ask for permission once the loop is running.
When to use
Use for an open-ended, autonomous ML research campaign where you want each change motivated by analysis
of the model's actual behaviour. Set <literature> = off for a self-contained analysis-and-score loop;
set <literature> = on to additionally ground changes in the scientific literature (paper search,
evidence grading, a reusable findings backlog). Not for a single training run, a fixed sweep, or tasks
with no measurable scalar metric. Default to off unless the user wants literature grounding or the
problem is a known, well-published one where prior recipes will pay off.
Setup
Resolve bindings interactively. If loop.run.yaml exists in the working dir, load it and skip to
the loop. Otherwise: on Claude Code (the AskUserQuestion tool is available — record <host> =
claude-code) infer a likely value for each binding from the project and present it as the recommended
option; on other hosts (<host> = other) ask each as a quoted plain-text prompt. Then write
loop.run.yaml (format: examples/run.example.yaml) and confirm every value with the user before
creating any other files. For branches strategy, create git checkout -b autoresearch/<run_tag>
(tag from today's date; branch must not exist). For time gating, write <sandbox_root>/run_with_timeout.sh
(timeout $(( <budget> * 60 )) <entrypoint> "$@") and use it as the run command, hard-killing at
2 × <budget> min; for epochs, patch the epoch cap in an <editable_files> file.
| binding | meaning | default | how to infer |
|---|---|---|---|
<metric> / <metric_direction> | scalar to optimize + minimize/maximize | — | scan editable files + README for metric names |
<run_cmd> / <entrypoint> | command that runs one experiment end to end | — | pyproject.toml / .venv / README |
<editable_files> | files fair game to edit (never the eval harness) | — | model / config / train scripts; exclude data, logs, env, harness |
<sandbox_root> | where snapshots + ledgers live | ./sandbox | next to the editable files |
<iter_strategy> | snapshots or branches | snapshots | is the working dir a clean git repo? |
<gate> / <budget> | time (min) or epochs, and the limit | — | existing time/epoch settings in config |
<literature> | on = literature-grounded; off = analysis-only | off | does the user want paper grounding? |
<research_scale> (on only) | depth dial low/medium/high/x-high | medium | see roles/research-subagent.md |
<domain> (on only) | one-phrase problem domain; seeds query phrasing only, never filters | — | infer from data/model/task |
<lit_skill_dir> (on only) | install dir of the literature-search skill | sibling of this loop | ~/.claude/skills/literature-search/ (adjust per host) |
<lit_py> (on only) | Python ≥3.9 interpreter for the lit helper (stdlib-only) | python3 | independent of <run_cmd> |
FILE EDIT GUARD: before touching any file at any point — setup or loop — confirm it is in
<editable_files>, because everything else is read-only ground truth (the eval harness defines
<metric>). No exceptions.
Literature toolchain (only when <literature> = on)
Paper search goes through the sibling literature-search skill (stdlib-only, no installs):
<lit> = <lit_skill_dir>/tools/lit_search.py (note the tools/ segment). Reuse one cache by
appending --cache-dir <sandbox_root>/literature/.cache after the subcommand. Subcommands print JSON;
on failure they print {"error","fallback"} and exit non-zero — then degrade to the host's
WebSearch/WebFetch (never fabricate citations). Smoke-test <lit> --help at setup; if the skill is
absent, tell the user and offer to install it (cp -r <repo>/loops/literature-search ~/.claude/skills/)
or proceed degraded. For onboarding and API keys (all optional; a free S2_API_KEY is recommended),
run <lit> keys --init — it manages the shared gitignored keys.env at the project root and reports
presence as booleans (secrets never enter chat). Persist the live tiers to loop.run.yaml.
Initialise the sandbox
Create the layout (extra literature/ tree only when <literature> = on) and write the ledger headers:
<sandbox_root>/
├── loop.run.yaml ← resolved bindings (written now)
├── results.tsv ← experiment ledger, header only (written now)
├── literature/ ← (on only)
│ ├── corpus.tsv ← findings ledger, header only (written now)
│ ├── .cache/ ← lit_search on-disk cache
│ ├── pdfs/ ← fallback PDF reads
│ └── text/ ← extracted LaTeX section text
└── iter1/ ← created
阅读完整来源 ↗