draco-analyze
Diagnose a commit's DRACO benchmark runs and connect each finding back to the atlas pipeline so the user gets root causes and concrete fixes, not just scores.
steel-dev · Static checks on file
Installation
Read the source and dependencies before running a command. Copying does not install or execute anything.
npx skills add steel-dev/atlas --skill draco-analyzeAbout this skill
name: draco-analyze description: Diagnose the DRACO benchmark results for an atlas commit — figure out why cases scored as they did and propose concrete atlas improvements, drilling from a commit overview down to per-criterion grades, claims, sources, and the byte-exact model transcript. Use when the user asks to "analyze this commit's benchmark", investigate a score regression, or work out why a DRACO case failed or underperformed. Read-only: it inspects stored runs, it does not run the benchmark. user-invocable: true allowed-tools: Bash, Read, Grep, Glob
draco-analyze
Diagnose a commit's DRACO benchmark runs and connect each finding back to the atlas pipeline so the user gets root causes and concrete fixes, not just scores.
Every run the explorer records is fully visible: the per-criterion judge grades, the produced report, the full claim ledger (with quotes, source provenance, and each verifier's vote), every fetched source's body, the pipeline event timeline, and the byte-exact transcript of every model step (lead, verifiers, leaf sub-agents) — what each model saw and the thinking + tool calls it produced. Runs are append-only, so repeated runs of the same case all survive.
All of this is read through one keyless CLI (evals/explore/query.ts). The data
is large, so the golden rule is progressive disclosure: start from the cheap
overview and descend only as far as a given question needs. Never dump a full
transcript blind — slice it.
Step 0 — Confirm the repo and find the DB
test -f evals/explore/query.ts && test -f package.json && echo ok || echo not-atlas
If not-atlas, ask the user to cd into the atlas repo and re-run. The DB
defaults to eval-runs/draco-explore.db; pass --db <path> to every command if
the user names a different one. (Commands print a harmless
ExperimentalWarning: SQLite to stderr — ignore it, or append 2>/dev/null.)
The query CLI is invoked as npx tsx evals/explore/query.ts <command> …. Run it
with no args for the full command list. Output is JSON; parse it, don't just echo
it at the user.
Step 1 — Resolve the commit and read the overview
If the user gave a commit SHA, use it. Otherwise default to the checked-out one
(git rev-parse HEAD) and say which you used.
npx tsx evals/explore/query.ts commits # all commits with runs
npx tsx evals/explore/query.ts commit <sha> # per-case grid, Δ vs previous commit
commit <sha> auto-compares against the previous commit; pass
--baseline <otherSha> to compare against a specific one (e.g. a known-good
reference). Read off:
regressions— cases whose score dropped vs the baseline. These are usually where to start.- per-case
normalized,passRate,failedCriteria(count),status(scored/error/unrun),judgeErrors.
Pick the highest-value targets: the biggest regressions, hard errors, and unexpectedly low scores. State your shortlist before drilling.
Step 2 — Diagnose each target case
npx tsx evals/explore/query.ts case <sha> <caseId>
This is the richest single view. It fuses the rubric with the judge's grades and the run's evidence:
failedCriteria— exactly which rubric criteria got UNMET, with the judge'sreason. This is your primary signal for what went wrong.score,claimStats(confirmed / refuted / unverified),diagnostics(fetch/claim health),finishReason(did it run out of budget? time?).runs— every recorded run of this case (append-only). If scores vary across identical runs, that's variance, not a deterministic bug — note it.runId+artifacts— the run to drill into and what it stored.
Step 3 — Turn symptoms into hypotheses
Map what you see to the likely pipeline stage, then drill the matching evidence.
Symptom in case | Likely cause | Drill with | atlas source |
|---|---|---|---|
| Factual criterion UNMET; few confirmed, many refuted | weak retrieval or over-strict verification | claims --status refuted, sources | verify.ts, fetch-tool.ts |
diagnostics shows many blockedOrThin sources | fetch/extraction failing | sources --blocked, trace --grep fetch | fetch-tool.ts, html-extract.ts |
| Coverage/depth criterion UNMET; report thin | lead stopped early / pursued wrong angles | transcript --role lead | research-loop.ts, recall.ts |
finishReason = budget/timeout | run starved before finishing | transcript summary (step count, in= tokens) | config-resolution.ts, runtime.ts |
Judge reason contradicts the report | possible judge error | rubric cross-check (Step 4) | grading in draco.ts |
| A claim wrongly refuted | a verifier over-refused | transcript --grep <claim text> → role verify:<id> | verify.ts |
Evidence commands (all take a <runId> from Step 2):
npx tsx evals/explore/query.ts claims <runId> [--status refuted] # quotes, sourceId, per-vote evidence
npx tsx evals/explore/query.ts sources <runId> [--blocked] # fetched sources; --id <s> dumps a body
npx tsx evals/explore/query.ts citations <runId> # what the report cited; not-fetched/-confirmed
npx tsx evals/explore/query.ts trace <runId> [--grep RE] # pipeline event timeline
npx tsx evals/explore/query.ts diagnostics <runId> # aggregate health counters
Step 4 — Cross-check the report against the rubric yourself
Don't take the judge's word for it. Read the produced report and the rubric and grade the failed criteria independently:
npx tsx evals/explore/query.ts report <runId> # the produced markdown
npx tsx evals/explore/query.ts rubric <caseId> # sections, criteria, weights
For each UNMET criterion decide which it is — the distinction drives a different fix:
- Content genuinely missing → a research/coverage problem (retrieval, lead loop, or budget).
- **Content present but not cr