跳至正文
OpenAgentSkill
浏览已开放 · 目录记录日期 2026-11-21 · 自动更新和账户服务暂时关闭。
浏览器与自动化

draco-analyze

Diagnose a commit's DRACO benchmark runs and connect each finding back to the atlas pipeline so the user gets root causes and concrete fixes, not just scores.

steel-dev · 已有静态检查记录

安装与使用

运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。

npx skills add steel-dev/atlas --skill draco-analyze

技能说明


name: draco-analyze description: Diagnose the DRACO benchmark results for an atlas commit — figure out why cases scored as they did and propose concrete atlas improvements, drilling from a commit overview down to per-criterion grades, claims, sources, and the byte-exact model transcript. Use when the user asks to "analyze this commit's benchmark", investigate a score regression, or work out why a DRACO case failed or underperformed. Read-only: it inspects stored runs, it does not run the benchmark. user-invocable: true allowed-tools: Bash, Read, Grep, Glob

draco-analyze

Diagnose a commit's DRACO benchmark runs and connect each finding back to the atlas pipeline so the user gets root causes and concrete fixes, not just scores.

Every run the explorer records is fully visible: the per-criterion judge grades, the produced report, the full claim ledger (with quotes, source provenance, and each verifier's vote), every fetched source's body, the pipeline event timeline, and the byte-exact transcript of every model step (lead, verifiers, leaf sub-agents) — what each model saw and the thinking + tool calls it produced. Runs are append-only, so repeated runs of the same case all survive.

All of this is read through one keyless CLI (evals/explore/query.ts). The data is large, so the golden rule is progressive disclosure: start from the cheap overview and descend only as far as a given question needs. Never dump a full transcript blind — slice it.

Step 0 — Confirm the repo and find the DB

test -f evals/explore/query.ts && test -f package.json && echo ok || echo not-atlas

If not-atlas, ask the user to cd into the atlas repo and re-run. The DB defaults to eval-runs/draco-explore.db; pass --db <path> to every command if the user names a different one. (Commands print a harmless ExperimentalWarning: SQLite to stderr — ignore it, or append 2>/dev/null.)

The query CLI is invoked as npx tsx evals/explore/query.ts <command> …. Run it with no args for the full command list. Output is JSON; parse it, don't just echo it at the user.

Step 1 — Resolve the commit and read the overview

If the user gave a commit SHA, use it. Otherwise default to the checked-out one (git rev-parse HEAD) and say which you used.

npx tsx evals/explore/query.ts commits                      # all commits with runs
npx tsx evals/explore/query.ts commit <sha>                 # per-case grid, Δ vs previous commit

commit <sha> auto-compares against the previous commit; pass --baseline <otherSha> to compare against a specific one (e.g. a known-good reference). Read off:

  • regressions — cases whose score dropped vs the baseline. These are usually where to start.
  • per-case normalized, passRate, failedCriteria (count), status (scored / error / unrun), judgeErrors.

Pick the highest-value targets: the biggest regressions, hard errors, and unexpectedly low scores. State your shortlist before drilling.

Step 2 — Diagnose each target case

npx tsx evals/explore/query.ts case <sha> <caseId>

This is the richest single view. It fuses the rubric with the judge's grades and the run's evidence:

  • failedCriteria — exactly which rubric criteria got UNMET, with the judge's reason. This is your primary signal for what went wrong.
  • score, claimStats (confirmed / refuted / unverified), diagnostics (fetch/claim health), finishReason (did it run out of budget? time?).
  • runs — every recorded run of this case (append-only). If scores vary across identical runs, that's variance, not a deterministic bug — note it.
  • runId + artifacts — the run to drill into and what it stored.

Step 3 — Turn symptoms into hypotheses

Map what you see to the likely pipeline stage, then drill the matching evidence.

Symptom in caseLikely causeDrill withatlas source
Factual criterion UNMET; few confirmed, many refutedweak retrieval or over-strict verificationclaims --status refuted, sourcesverify.ts, fetch-tool.ts
diagnostics shows many blockedOrThin sourcesfetch/extraction failingsources --blocked, trace --grep fetchfetch-tool.ts, html-extract.ts
Coverage/depth criterion UNMET; report thinlead stopped early / pursued wrong anglestranscript --role leadresearch-loop.ts, recall.ts
finishReason = budget/timeoutrun starved before finishingtranscript summary (step count, in= tokens)config-resolution.ts, runtime.ts
Judge reason contradicts the reportpossible judge errorrubric cross-check (Step 4)grading in draco.ts
A claim wrongly refuteda verifier over-refusedtranscript --grep <claim text> → role verify:<id>verify.ts

Evidence commands (all take a <runId> from Step 2):

npx tsx evals/explore/query.ts claims <runId> [--status refuted]   # quotes, sourceId, per-vote evidence
npx tsx evals/explore/query.ts sources <runId> [--blocked]          # fetched sources; --id <s> dumps a body
npx tsx evals/explore/query.ts citations <runId>                    # what the report cited; not-fetched/-confirmed
npx tsx evals/explore/query.ts trace <runId> [--grep RE]            # pipeline event timeline
npx tsx evals/explore/query.ts diagnostics <runId>                  # aggregate health counters

Step 4 — Cross-check the report against the rubric yourself

Don't take the judge's word for it. Read the produced report and the rubric and grade the failed criteria independently:

npx tsx evals/explore/query.ts report <runId>     # the produced markdown
npx tsx evals/explore/query.ts rubric <caseId>    # sections, criteria, weights

For each UNMET criterion decide which it is — the distinction drives a different fix:

  • Content genuinely missing → a research/coverage problem (retrieval, lead loop, or budget).
  • **Content present but not cr
阅读完整来源 ↗