跳至正文
OpenAgentSkill
浏览已开放 · 目录记录日期 2026-11-21 · 自动更新和账户服务暂时关闭。
开发与测试

weak-agent-test

Run the weak-agent adversarial test harness against docx-cli. Spawns weak exercise agents (Haiku by default, Sonnet to probe, or a local agent harness's pre-produced runs) to perform real document tasks over six scenarios — five editing (MNDA form-fill + font fidelity, invoice table-edit/restructure + logo replace, résumé styling, contract redlining + commenting, contract finalize via accept/reject + comment reply/resolve) and one authoring (T. S. Eliot poetry journal: multi-column, verse, footnotes, links, figure) — renders every result with Word, has opus judge them against ground-truth rubrics, measures each exercise's tool economy, token cost, wall-clock, and correctness (from transcripts for Claude, the exercise.json ledger for the local harness), and synthesizes a prioritized ergonomics report. Use when the user says 'adversarial review', 'test docx-cli with weak agents', 'run the haiku harness', 'weak agent test', or wants to re-run yesterday's adversarial process.

kklimuk · 已有 AI 审核记录

安装与使用

运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。

npx skills add kklimuk/docx-cli --skill weak-agent-test

技能说明


name: weak-agent-test description: "Run the weak-agent adversarial test harness against docx-cli. Spawns weak exercise agents (Haiku by default, Sonnet to probe, or a local agent harness's pre-produced runs) to perform real document tasks over six scenarios — five editing (MNDA form-fill + font fidelity, invoice table-edit/restructure + logo replace, résumé styling, contract redlining + commenting, contract finalize via accept/reject + comment reply/resolve) and one authoring (T. S. Eliot poetry journal: multi-column, verse, footnotes, links, figure) — renders every result with Word, has opus judge them against ground-truth rubrics, measures each exercise's tool economy, token cost, wall-clock, and correctness (from transcripts for Claude, the exercise.json ledger for the local harness), and synthesizes a prioritized ergonomics report. Use when the user says 'adversarial review', 'test docx-cli with weak agents', 'run the haiku harness', 'weak agent test', or wants to re-run yesterday's adversarial process." allowed-tools: Bash, Read, Write, Glob, Workflow metadata: internal: true

Adversarial review — weak-agent harness for docx-cli

This harness answers one question: can weak agents actually use docx-cli to get real work done, and what should we fix first? It runs the weak-agent-test workflow (.claude/workflows/weak-agent-test.js), which fans out one weak exercise agent per scenario (Haiku by default — swappable to Sonnet via args.model), renders every output with Microsoft Word, grades each against ground-truth criteria with an opus judge, and has opus synthesize a prioritized improvement report. Exercise agents do NOT self-report tool counts — every tool-economy and token number is measured after the run (agents under-count their own calls ~2×, so self-reports were dropped): from the agent transcripts for the Claude arms, from each scenario's exercise.json ledger for the local arm. Both roll up into the same Run-metrics table (tokens, wall-clock, tool split, correctness) via exercise-metrics.ts.

The test corpus is bundled with this skill under scenarios/, one folder per scenario, named after its key (scenarios/mnda/, scenarios/invoice/, …). Each scenario folder is self-describing and holds everything that scenario needs:

  • task.md — the AGENT-FACING request, written as a human delegating the work: the goal, the data, the intent — and no tool vocabulary (no docx commands, locators, or OOXML terms), because discovering which features deliver the outcome is part of what's measured,
  • criteria.md — the JUDGE-ONLY grading rubric (the precise, tool-specific checks). The stage step withholds it from the agent's run workspace, and the judge reads it from the pristine source — the agent never sees the answer key,
  • the fixture .docx to work on (edit scenarios only; authoring scenarios create their output fresh),
  • assets/ — any additional inputs (data files, images; empty for most edit scenarios).

The workflow's SCENARIOS manifest holds only the per-scenario routing metadata (key, bucket label, edit/author kind, the doc filename); whether a baseline gets rendered is DERIVED from the kind (every edit scenario has a pristine source, so it gets one — see hasBaseline()), not a stored field. The actual request/criteria/fixture/assets all live in the folder. The skill is therefore self-contained and travels with its test corpus. To change what a scenario tests, edit the files in its folder. (Heavy, ephemeral run outputs — edited docx, renders, reviews, the report — are dumped to ./tmp/docx-weak-agent-test/<ts>/, never into the repo.)

Staging is ONE code path for every backend: scripts/stage-scenario.ts copies a scenario folder, strips the judge-only criteria.md, and verifies the inputs landed. The workflow's Stage agent runs it per scenario; the local corpus runner imports it.

Each run produces, under the timestamped run dir, one result folder per scenario (named after its key) plus the run-level report and metrics:

<RUN_DIR>/
  REPORT.md            ← synthesized report; the Metrics phase appends the measured
                          run-metrics section (local: in-run; Claude: your post-run pass)
  exercise-metrics.md  ← measured per-exercise-agent tokens/time/tool split
  exercise-metrics.json
  <key>/               ← one per scenario; the worked-on copy lives here
    task.md  assets/   ← (criteria.md is withheld from this copy — judge-only)
    <doc>.docx         ← the edited/authored document
    renders/output/    ← the OUTPUT: Word-rendered page PNGs + read.md (markdown read view)
    renders/baseline/  ← the pristine "before": page PNGs + read.md (every EDIT scenario;
                          absent only for the authored eliot-journal — no source to diff)
    review.md          ← the judge's saved review for this task (written in-run)
    verdict.json       ← the judge's structured verdict incl. taskSuccess (written in-run
                          by the judge — the correctness source the Metrics phase reads)
    metrics.json       ← this task's measured tokens/time/tool split + correctness
                          (local: in-run Metrics phase; Claude: your post-run pass)

The render step fires the moment each task finishes (for both arms) and produces, for the OUTPUT and — whenever a pristine source exists (every edit scenario) — its BASELINE "before", BOTH deliverables in each render dir: the page PNGs AND a read.md (the markdown read view of that doc). The judge reads all four (output PNGs + read.md, baseline PNGs + read.md) to compare before/after both visually and textually. The workflow's render step is idempotent: for the local backend the corpus runner already produced the SAME artifacts at the SAME paths as it went, so the render step just reuses them (re-rendering only anything missing) — no double-render; for the Claude backend nothing is pre-

阅读完整来源 ↗