ai-agents-diagnostics-toolkit
Catalog of this repo's measurement instruments, each with command, current baseline, and interpretation guide. Covers skill size and description budgets, orphan-ref and golden-principles scans, drift gates as signals, coverage pins, and the eval harness. Use when you say `measure this`, `read the drift signal`, `check skill budgets`. Do NOT use to fix what you measure (use `ai-agents-debugging-playbook`) or for evidence standards (use `ai-agents-validation-and-qa`).
rjmurillo · 已有静态检查记录
安装与使用
运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。
npx skills add rjmurillo/ai-agents --skill ai-agents-diagnostics-toolkit技能说明
name: ai-agents-diagnostics-toolkit
version: 1.0.0
license: MIT
description: Catalog of this repo's measurement instruments, each with command, current baseline, and interpretation guide. Covers skill size and description budgets, orphan-ref and golden-principles scans, drift gates as signals, coverage pins, and the eval harness. Use when you say measure this, read the drift signal, check skill budgets. Do NOT use to fix what you measure (use ai-agents-debugging-playbook) or for evidence standards (use ai-agents-validation-and-qa).
metadata:
routing:
role: conditional-adjunct
invoker: ai-agents-debugging-playbook
trigger: ai-agents-debugging-playbook points here to measure drift, budgets, or telemetry as a number
user-facing: false
ai-agents Diagnostics Toolkit
Measure instead of eyeball. Every instrument below leaves the trees it measures unchanged, turning a vague worry ("are skills getting bloated?", "did generation drift?") into a number you can compare against a baseline; build_all.py --check's one exception, a gitignored audit file outside those trees, is covered below. The Instrument Index gives you, per instrument, the question it answers and the exact command; references/instrument-guides.md gives the healthy and unhealthy reading, the current repo baseline (as of 2026-07-29), and the trap that has already cost someone time.
Vocabulary, defined once: an "instrument" is a script whose output you read, not a gate you must pass. A "drift gate" is a CI check that fails when a generated tree stops matching its canonical source. A "baseline" is the number the instrument reports on a clean checkout of main; you measure your delta against it.
Triggers
measure thisread the drift signalcheck skill budgetsinterpret this scan output
Instrument Index
| Instrument | Question it answers | Command (from repo root) |
|---|---|---|
| Description budget | How much standing context do skill descriptions cost? | uv run python ./scripts/skill_description_budget.py |
| Skill size | Which SKILL.md files exceed the 300-warn / 500-block line limits? | uv run python ./scripts/validation/skill_size.py |
| Orphan refs | Do specs, evals, and manifests reference entities that no longer exist? | python3 "${COPILOT_PLUGIN_ROOT:-${CLAUDE_PLUGIN_ROOT:-.claude}}/skills/orphan-ref-validator/scripts/scan.py" |
| Golden principles | Where does the repo violate GP-001..GP-005 mechanical rules? | python3 "${COPILOT_PLUGIN_ROOT:-${CLAUDE_PLUGIN_ROOT:-.claude}}/skills/golden-principles/scripts/scan_principles.py" |
| Agent drift | Do generated agent files match their templates? | uv run python build/generate_agents.py --validate |
| Mirror drift | Do the 7 generated mirror trees match .claude/ canonical sources? | uv run python build/scripts/build_all.py --check |
| Lib drift | Do the lib plugin trees match scripts/ canonical modules? | uv run python build/scripts/build_all.py --check |
| Coverage | Is changed code actually exercised by tests? | uv run pytest <tests> --cov=<module> --cov-branch |
| Eval A/B | Did a prompt or agent change alter behavior, measurably? | uv run python ./scripts/eval/eval-prompt-change.py --scenarios <file> --dry-run |
| Commit count | Advisory-only signal, not a cap (ADR-099) | git rev-list --count HEAD ^origin/main |
Process
Phase 1: Pick the instrument
Match the worry to the row in the Instrument Index. Two routing rules:
| If you want to... | Go to |
|---|---|
| Fix the failure an instrument surfaced | ai-agents-debugging-playbook |
| Know what counts as test evidence | ai-agents-validation-and-qa |
| Prove a runtime hypothesis with a probe | ai-agents-empirical-probe-toolkit |
| Regenerate after a drift red | ai-agents-generation-and-release |
| Query agent JSONL event logs | observability skill (do not duplicate it) |
| Get adoption metrics from git history | metrics skill (collect_metrics.py) |
Phase 2: Run it correctly
- Run from the repo root. All commands above assume it.
- Use
uv run python, not barepython3, for anything that imports repo modules (PyYAML lives in the venv; barepython3givesModuleNotFoundError: No module named 'yaml'). - Every instrument here leaves the owned trees it reads unchanged in the modes shown,
build_all.py --checkincluded, but that one is not a no-write run overall: it runs its generators and then restores the owned trees from a snapshot (issue #2440,_snapshot_owned_prefixesand_restore_owned_prefixesinbuild/scripts/build_all.py), so its log printsMode: Generateand a nonzeroWritten:count even though the owned trees end the run as they started, and separatelywrite_auditcreates or overwritesbuild/audit/GENERATION-AUDIT.mdon every invocation, a path outside the snapshot that restore never touches. It is gitignored (/build/audit/), which is whygit status --porcelainstill comes back clean over the owned trees. Confirm withgit status --porcelainif suspicious, and see the trap in the Drift gates section ofreferences/instrument-guides.md. - Read the exit code, not just the prose. It is the machine signal:
| Exit code | Convention (ADR-035 / AGENTS.md) | Exceptions |
|---|---|---|
| 0 | Healthy or within limits | skill_size.py prints FAIL lines but exits 0 unless --ci |
| 1 | Logic finding (budget exceeded, CRITICAL_FAIL, over limit in --ci) | |
| 2 | Config error (bad path, bad args); for build_all.py --check, staleness, a path under OWNED_PREFIXES that cannot be read or redirects (symlink or junction) or |