weak-agent-test
Run the weak-agent adversarial test harness against docx-cli. Spawns weak exercise agents (Haiku by default, Sonnet to probe, or a local agent harness's pre-produced runs) to perform real document tasks over six scenarios — five editing (MNDA form-fill + font fidelity, invoice table-edit/restructure + logo replace, résumé styling, contract redlining + commenting, contract finalize via accept/reject + comment reply/resolve) and one authoring (T. S. Eliot poetry journal: multi-column, verse, footnotes, links, figure) — renders every result with Word, has opus judge them against ground-truth rubrics, measures each exercise's tool economy, token cost, wall-clock, and correctness (from transcripts for Claude, the exercise.json ledger for the local harness), and synthesizes a prioritized ergonomics report. Use when the user says 'adversarial review', 'test docx-cli with weak agents', 'run the haiku harness', 'weak agent test', or wants to re-run yesterday's adversarial process.
kklimuk · 已有 AI 审核记录
安装与使用
运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。
npx skills add kklimuk/docx-cli --skill weak-agent-test技能说明
name: weak-agent-test description: "Run the weak-agent adversarial test harness against docx-cli. Spawns weak exercise agents (Haiku by default, Sonnet to probe, or a local agent harness's pre-produced runs) to perform real document tasks over six scenarios — five editing (MNDA form-fill + font fidelity, invoice table-edit/restructure + logo replace, résumé styling, contract redlining + commenting, contract finalize via accept/reject + comment reply/resolve) and one authoring (T. S. Eliot poetry journal: multi-column, verse, footnotes, links, figure) — renders every result with Word, has opus judge them against ground-truth rubrics, measures each exercise's tool economy, token cost, wall-clock, and correctness (from transcripts for Claude, the exercise.json ledger for the local harness), and synthesizes a prioritized ergonomics report. Use when the user says 'adversarial review', 'test docx-cli with weak agents', 'run the haiku harness', 'weak agent test', or wants to re-run yesterday's adversarial process." allowed-tools: Bash, Read, Write, Glob, Workflow metadata: internal: true
Adversarial review — weak-agent harness for docx-cli
This harness answers one question: can weak agents actually use docx-cli to get
real work done, and what should we fix first? It runs the weak-agent-test
workflow (.claude/workflows/weak-agent-test.js), which fans out one weak exercise
agent per scenario (Haiku by default — swappable to Sonnet via args.model), renders
every output with Microsoft Word, grades each against ground-truth criteria with an
opus judge, and has opus synthesize a prioritized improvement report.
Exercise agents do NOT self-report tool counts — every tool-economy and token number
is measured after the run (agents under-count their own calls ~2×, so self-reports
were dropped): from the agent transcripts for the Claude arms, from each scenario's
exercise.json ledger for the local arm. Both roll up into the same Run-metrics table
(tokens, wall-clock, tool split, correctness) via exercise-metrics.ts.
The test corpus is bundled with this skill under scenarios/, one folder per
scenario, named after its key (scenarios/mnda/, scenarios/invoice/, …). Each
scenario folder is self-describing and holds everything that scenario needs:
task.md— the AGENT-FACING request, written as a human delegating the work: the goal, the data, the intent — and no tool vocabulary (nodocxcommands, locators, or OOXML terms), because discovering which features deliver the outcome is part of what's measured,criteria.md— the JUDGE-ONLY grading rubric (the precise, tool-specific checks). The stage step withholds it from the agent's run workspace, and the judge reads it from the pristine source — the agent never sees the answer key,- the fixture
.docxto work on (edit scenarios only; authoring scenarios create their output fresh), assets/— any additional inputs (data files, images; empty for most edit scenarios).
The workflow's SCENARIOS manifest holds only the per-scenario routing metadata
(key, bucket label, edit/author kind, the doc filename); whether a baseline gets
rendered is DERIVED from the kind (every edit scenario has a pristine source, so it
gets one — see hasBaseline()), not a stored field. The actual
request/criteria/fixture/assets all live in the folder. The skill is
therefore self-contained and travels with its test corpus. To change what a scenario
tests, edit the files in its folder. (Heavy, ephemeral run outputs — edited docx,
renders, reviews, the report — are dumped to ./tmp/docx-weak-agent-test/<ts>/,
never into the repo.)
Staging is ONE code path for every backend: scripts/stage-scenario.ts copies a
scenario folder, strips the judge-only criteria.md, and verifies the inputs landed.
The workflow's Stage agent runs it per scenario; the local corpus runner imports it.
Each run produces, under the timestamped run dir, one result folder per scenario (named after its key) plus the run-level report and metrics:
<RUN_DIR>/
REPORT.md ← synthesized report; the Metrics phase appends the measured
run-metrics section (local: in-run; Claude: your post-run pass)
exercise-metrics.md ← measured per-exercise-agent tokens/time/tool split
exercise-metrics.json
<key>/ ← one per scenario; the worked-on copy lives here
task.md assets/ ← (criteria.md is withheld from this copy — judge-only)
<doc>.docx ← the edited/authored document
renders/output/ ← the OUTPUT: Word-rendered page PNGs + read.md (markdown read view)
renders/baseline/ ← the pristine "before": page PNGs + read.md (every EDIT scenario;
absent only for the authored eliot-journal — no source to diff)
review.md ← the judge's saved review for this task (written in-run)
verdict.json ← the judge's structured verdict incl. taskSuccess (written in-run
by the judge — the correctness source the Metrics phase reads)
metrics.json ← this task's measured tokens/time/tool split + correctness
(local: in-run Metrics phase; Claude: your post-run pass)
The render step fires the moment each task finishes (for both arms) and produces,
for the OUTPUT and — whenever a pristine source exists (every edit scenario) — its
BASELINE "before", BOTH deliverables in each render dir: the page PNGs AND a read.md
(the markdown read view of that doc). The judge reads all four (output PNGs + read.md,
baseline PNGs + read.md) to compare before/after both visually and textually. The
workflow's render step is idempotent: for the local backend the corpus runner
already produced the SAME artifacts at the SAME paths as it went, so the render step
just reuses them (re-rendering only anything missing) — no double-render; for the
Claude backend nothing is pre-