trace-harness-launch-failure
Empty session traces mean launch failure, not agent behavior — verify api_call_count/tool_call_count and stdout.txt before analyzing.
ashutoshsinghpr7 · 已有 AI 审核记录
安装与使用
运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。
npx skills add ashutoshsinghpr7/wikiskill --skill trace-harness-launch-failure技能说明
name: trace-harness-launch-failure description: "Empty session traces mean launch failure, not agent behavior — verify api_call_count/tool_call_count and stdout.txt before analyzing." version: 1.0.0 license: MIT platforms: [linux, macos, windows] metadata: hermes: tags: [traces, sessions, launch-failure, forensics] homepage: https://github.com/ashutoshsinghpr7/wikiskill
Trace Harness Launch Failure — empty sessions, scores from deliverables
Problem
Four training traces contain ONLY the initial task prompt: 1 message,
0 tool calls, 0 API calls, ~1 s duration. No agent reasoning, tool call, or
final answer anywhere — yet meta.json carries scores (0.0 / 1.0).
Root cause
Every recorded run died at launch: runs/<task>/stdout.txt shows
"HTTP 400: default is not a valid model ID" — the harness sent an invalid
model ID instead of the configured model, so the agent never executed. The
scores were produced by grading the on-disk deliverables against the
exact-match graders in tasks.json — not from any recorded agent behavior.
Fix
- Before analyzing "agent behavior", verify the session actually ran: check
api_call_count/tool_call_count/message_countin the trace and stdout.txt for launch errors. Empty trace == no behavioral evidence. - When traces are empty, reconstruct ground truth by diffing the on-disk
deliverable against the task's grader expectation
(
tasks.json→grader.expected), and read spec.md + input data to explain the score. - Treat prior-iteration behavioral claims as unverifiable until new traces confirm them; flag evidence status in the run log.
- Surface the harness bug so the runner is fixed before the next iteration, otherwise every proposal rests on inferred data.