跳至正文
OpenAgentSkill
浏览已开放 · 目录记录日期 2026-11-21 · 自动更新和账户服务暂时关闭。
其他技能

ai-agents-diagnostics-toolkit

Catalog of this repo's measurement instruments, each with command, current baseline, and interpretation guide. Covers skill size and description budgets, orphan-ref and golden-principles scans, drift gates as signals, coverage pins, and the eval harness. Use when you say `measure this`, `read the drift signal`, `check skill budgets`. Do NOT use to fix what you measure (use `ai-agents-debugging-playbook`) or for evidence standards (use `ai-agents-validation-and-qa`).

rjmurillo · 已有静态检查记录

安装与使用

运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。

npx skills add rjmurillo/ai-agents --skill ai-agents-diagnostics-toolkit

技能说明


name: ai-agents-diagnostics-toolkit version: 1.0.0 license: MIT description: Catalog of this repo's measurement instruments, each with command, current baseline, and interpretation guide. Covers skill size and description budgets, orphan-ref and golden-principles scans, drift gates as signals, coverage pins, and the eval harness. Use when you say measure this, read the drift signal, check skill budgets. Do NOT use to fix what you measure (use ai-agents-debugging-playbook) or for evidence standards (use ai-agents-validation-and-qa). metadata: routing: role: conditional-adjunct invoker: ai-agents-debugging-playbook trigger: ai-agents-debugging-playbook points here to measure drift, budgets, or telemetry as a number user-facing: false

ai-agents Diagnostics Toolkit

Measure instead of eyeball. Every instrument below leaves the trees it measures unchanged, turning a vague worry ("are skills getting bloated?", "did generation drift?") into a number you can compare against a baseline; build_all.py --check's one exception, a gitignored audit file outside those trees, is covered below. The Instrument Index gives you, per instrument, the question it answers and the exact command; references/instrument-guides.md gives the healthy and unhealthy reading, the current repo baseline (as of 2026-07-29), and the trap that has already cost someone time.

Vocabulary, defined once: an "instrument" is a script whose output you read, not a gate you must pass. A "drift gate" is a CI check that fails when a generated tree stops matching its canonical source. A "baseline" is the number the instrument reports on a clean checkout of main; you measure your delta against it.

Triggers

  • measure this
  • read the drift signal
  • check skill budgets
  • interpret this scan output

Instrument Index

InstrumentQuestion it answersCommand (from repo root)
Description budgetHow much standing context do skill descriptions cost?uv run python ./scripts/skill_description_budget.py
Skill sizeWhich SKILL.md files exceed the 300-warn / 500-block line limits?uv run python ./scripts/validation/skill_size.py
Orphan refsDo specs, evals, and manifests reference entities that no longer exist?python3 "${COPILOT_PLUGIN_ROOT:-${CLAUDE_PLUGIN_ROOT:-.claude}}/skills/orphan-ref-validator/scripts/scan.py"
Golden principlesWhere does the repo violate GP-001..GP-005 mechanical rules?python3 "${COPILOT_PLUGIN_ROOT:-${CLAUDE_PLUGIN_ROOT:-.claude}}/skills/golden-principles/scripts/scan_principles.py"
Agent driftDo generated agent files match their templates?uv run python build/generate_agents.py --validate
Mirror driftDo the 7 generated mirror trees match .claude/ canonical sources?uv run python build/scripts/build_all.py --check
Lib driftDo the lib plugin trees match scripts/ canonical modules?uv run python build/scripts/build_all.py --check
CoverageIs changed code actually exercised by tests?uv run pytest <tests> --cov=<module> --cov-branch
Eval A/BDid a prompt or agent change alter behavior, measurably?uv run python ./scripts/eval/eval-prompt-change.py --scenarios <file> --dry-run
Commit countAdvisory-only signal, not a cap (ADR-099)git rev-list --count HEAD ^origin/main

Process

Phase 1: Pick the instrument

Match the worry to the row in the Instrument Index. Two routing rules:

If you want to...Go to
Fix the failure an instrument surfacedai-agents-debugging-playbook
Know what counts as test evidenceai-agents-validation-and-qa
Prove a runtime hypothesis with a probeai-agents-empirical-probe-toolkit
Regenerate after a drift redai-agents-generation-and-release
Query agent JSONL event logsobservability skill (do not duplicate it)
Get adoption metrics from git historymetrics skill (collect_metrics.py)

Phase 2: Run it correctly

  • Run from the repo root. All commands above assume it.
  • Use uv run python, not bare python3, for anything that imports repo modules (PyYAML lives in the venv; bare python3 gives ModuleNotFoundError: No module named 'yaml').
  • Every instrument here leaves the owned trees it reads unchanged in the modes shown, build_all.py --check included, but that one is not a no-write run overall: it runs its generators and then restores the owned trees from a snapshot (issue #2440, _snapshot_owned_prefixes and _restore_owned_prefixes in build/scripts/build_all.py), so its log prints Mode: Generate and a nonzero Written: count even though the owned trees end the run as they started, and separately write_audit creates or overwrites build/audit/GENERATION-AUDIT.md on every invocation, a path outside the snapshot that restore never touches. It is gitignored (/build/audit/), which is why git status --porcelain still comes back clean over the owned trees. Confirm with git status --porcelain if suspicious, and see the trap in the Drift gates section of references/instrument-guides.md.
  • Read the exit code, not just the prose. It is the machine signal:
Exit codeConvention (ADR-035 / AGENTS.md)Exceptions
0Healthy or within limitsskill_size.py prints FAIL lines but exits 0 unless --ci
1Logic finding (budget exceeded, CRITICAL_FAIL, over limit in --ci)
2Config error (bad path, bad args); for build_all.py --check, staleness, a path under OWNED_PREFIXES that cannot be read or redirects (symlink or junction) or
阅读完整来源 ↗