Skip to content
OpenAgentSkill
Browsing is open · Catalog captured 2026-11-21 · Automatic updates and account services are paused.
Search & research

autoresearch

Autonomous skill-prompt optimization — Karpathy-style mutate/score/keep loop on SKILL.md. Triggers "autoresearch", "optimize skill", "tune", "evolve" a skill, "prompt optimization".

darkroomengineering · AI review on file

Installation

Read the source and dependencies before running a command. Copying does not install or execute anything.

npx skills add darkroomengineering/cc-settings --skill autoresearch

About this skill


name: autoresearch description: Autonomous skill-prompt optimization — Karpathy-style mutate/score/keep loop on SKILL.md. Triggers "autoresearch", "optimize skill", "tune", "evolve" a skill, "prompt optimization". context: fork argument-hint: "[skill-name]"

AutoResearch

Autonomous skill optimization. You modify a skill's prompt, test it, keep improvements, revert failures. Repeat forever.

Adapted from Karpathy's autoresearch. Same method: single editable file, single metric, git-based keep/revert, autonomous loop. The only difference: SKILL.md replaces train.py, checklist pass rate replaces val_bpb.

NEVER STOP. Once the loop begins, do NOT pause to ask the human if you should continue. The human might be away and expects you to work indefinitely until manually interrupted. If you run out of ideas, think harder — re-read failing outputs, try combining near-misses, try more radical prompt rewrites. The loop runs until the human interrupts you, period.


Setup

Work with the user to configure, then go autonomous.

  1. Parse target skill: Get <skill-name> from $ARGUMENTS. Validate skills/<skill-name>/SKILL.md exists.

  2. Load or create RESEARCH.md: Check for skills/<skill-name>/RESEARCH.md. If it exists, read it — a skill born from /harvest arrives with a seeded RESEARCH.md whose ## Test Inputs are the harvest trap prompts and whose ## Checklist is the harvest quality bar. If not, generate one:

    • Read the target SKILL.md
    • Derive 3 test inputs from its description and use cases
    • Derive 5-7 checklist items from its workflow steps and output format
    • Write the generated RESEARCH.md and show it to the user for confirmation

    Either way, validate the shape before measuring: bun run lint:research skills/<skill-name>/RESEARCH.md (required sections present, ≥2 test inputs, 3-7 checklist items, numeric settings). A seed that fails this parses wrong in the loop below.

  3. Parse config from RESEARCH.md:

    • ## Test Inputs — each ### Test N: heading is one test case (the text below is the prompt)
    • ## Checklist — each - [ ] line is a binary criterion
    • ## Settings — optional: samples (default 3), min_improvement (default 0.05), max_rounds (default 50), model (default claude-sonnet-5, exported as AUTORESEARCH_MODEL — see Sample Isolation)
  4. Create results directory:

    mkdir -p ~/.claude/tmp/autoresearch/<skill-name>
    
  5. Initialize results.tsv:

    echo -e "round\tcommit\tscore\tcorrectness\tsafety\tsamples\tstatus\tdescription" > ~/.claude/tmp/autoresearch/<skill-name>/results.tsv
    
  6. Create branch: git checkout -b autoresearch/<skill-name> from current HEAD. If the branch already exists, check it out and resume (read existing results.tsv for history).

  7. Read the SKILL.md as the baseline prompt. Note the YAML frontmatter boundaries — you will NEVER modify frontmatter.

  8. Confirm and go: Show the user the config summary (target, test count, checklist count, samples per round, pinned model). Get confirmation. Then go autonomous.


Baseline

Before any mutations, measure the starting score.

  1. Run N samples (N = samples from settings):

    • For each sample, pick a test input (cycle through test inputs round-robin)
    • Run the sample in an isolated session — see Sample Isolation below. Never spawn an in-process Agent(...) for a sample.
    • Capture stdout as the sample output
  2. Score each output using the Scoring Protocol (below).

  3. Compute mean score across all samples, plus mean_correctness and mean_safety.

  4. Log to results.tsv:

    0	baseline	{score}	{correctness}	{safety}	{N}	baseline	initial measurement
    
  5. Print: Baseline score: {score} ({X}/{Y} checklist items passing on average) · correctness {c}/5 · safety {s}/5 · model {AUTORESEARCH_MODEL}

  6. Set best_score = score, baseline_correctness = mean_correctness, baseline_safety = mean_safety. These two are the floor for every later round and never move, even when a mutation improves them — a later regression is measured against the original skill, not against the best round so far. Begin the loop.


Sample Isolation

A sample run must not inherit this machine's configuration. An in-process Agent(...) call loads ~/.claude/CLAUDE.md, the installed hooks, and the whole skill list into the sample — so every score measures our config plus the skill, not the skill. When the skill under test overlaps anything in CLAUDE.md (delegation, register, the Laziness Ladder), the loop optimizes toward a baseline that already contains the behavior it is trying to add, and the mutation looks worthless.

Run every sample as a subprocess with settings disabled and the model pinned:

# Strip YAML frontmatter, keep the body — the frontmatter is never under test.
BODY=$(awk 'NR==1 && /^---$/ {fm=1; next} fm && /^---$/ {fm=0; next} !fm' \
  "skills/<skill-name>/SKILL.md")

claude -p \
  --setting-sources "" \
  --strict-mcp-config \
  --model "$AUTORESEARCH_MODEL" \
  --append-system-prompt "$BODY" \
  "<test input>"
  • --setting-sources "" loads none of user, project, local. Without it the operator's CLAUDE.md, hooks, memory, and output style leak into every condition.
  • --strict-mcp-config keeps MCP servers out unless the skill declares them.
  • --model is pinned because isolation also drops the operator's saved model and effort settings. Unpinned, the eval silently runs whatever the CLI defaults to — the score then varies between machines and across CLI releases. Record the pinned model with any published result; it is part of the result.

Set AUTORESEARCH_MODEL once at setup (default claude-sonnet-5) and never change it mid-run — a model swap invalidates every earlier row in results.tsv.

Control arm. Mutation scores are relative: they say varia

Read the complete source ↗