ai-agents-research-methodology
How a hunch becomes an accepted result in this repo. Covers the
rjmurillo · 已有静态检查记录
安装与使用
运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。
npx skills add rjmurillo/ai-agents --skill ai-agents-research-methodology技能说明
name: ai-agents-research-methodology
version: 1.0.0
license: MIT
description: How a hunch becomes an accepted result in this repo. Covers the
evidence bar, hypothesis-predicts-numbers discipline, and the idea lifecycle
from contradiction log through probe, eval baseline, ADR debate, calibrated
gate, and post-ship monitoring. Use when you say how do I prove this idea, run the idea lifecycle, what is the evidence bar. Do NOT use for
the open research programs (use ai-agents-research-frontier) or probe recipe
depth (use ai-agents-empirical-probe-toolkit).
metadata:
routing:
role: conditional-adjunct
invoker: ai-agents-external-claims
trigger: ai-agents-external-claims points here to prove a repo-internal idea with a probe, eval, or ADR debate
user-facing: false
AI Agents Research Methodology
This repo runs on verification-based governance: labels like "MANDATORY" or "NON-NEGOTIABLE" are insufficient, every requirement needs a verification mechanism. The same standard applies to ideas. An idea is not accepted because it sounds right, because a model agreed with it, or because a retro asserted it. It is accepted when it survives the lifecycle below and leaves an inspectable artifact at every stage.
This skill is the discipline. For the specific probe recipes, use
ai-agents-empirical-probe-toolkit. For the three open research programs, use
ai-agents-research-frontier. For the archive of past settled results, use
ai-agents-failure-archaeology.
Triggers
how do I prove this idearun the idea lifecyclewhat is the evidence barturn this hunch into a result
The Evidence Bar
A result is accepted here when ONE mechanism explains ALL observations, including the negative ones, and the explanation survives adversarial refutation. Partial explanations that cover only the confirming observations are hypotheses, not results.
The cautionary tale is PR #1989. Mitigation M1 was built on a root-cause claim
inherited from a retro RCA that nobody re-verified: the RCA said pagination was
missing, but get_unresolved_review_threads.py already paginated correctly
(.project-toolkit/retrospective/2026-05-10-pr-1989-recursive-failure.md:20). Five
commits were spent building on the false premise. The mechanism-check that
would have caught it costs one file read.
Map each claim type to the adversarial machinery that already exists. Do not invent a new review ritual; route into these:
| Claim type | Adversarial mechanism | How to invoke |
|---|---|---|
| Architectural decision (ADR) | adr-review 6-agent debate (architect, critic, independent-thinker, security, analyst, high-level-advisor) until consensus, 10 rounds max | Auto-fires on any ADR-*.md create/edit (AGENTS.md "ADR Review"); or /adr-review path |
| A single decision's reasoning | decision-critic skill | Say Poke holes in this decision or Validate my thinking on ... |
| Contrarian read of a plan | independent-thinker or critic agent (.claude/agents/) | Task tool with that subagent_type |
| Strategic build/buy/defer | buy-vs-build-framework (Quick tier: 1-2 hours) | Required gate for new capabilities, see Phase 3 |
| "Why does this constraint exist" | chestertons-fence | Before proposing removal of anything settled |
| Behavioral claim about a prompt or rule | Eval harness (ADR-057) | See Predict Numbers Before Running |
| Prose claims in the write-up | prose-self-check, doc-accuracy | Before emitting the artifact |
Predict Numbers Before Running
Write the predicted outcome down BEFORE running the measurement. A prediction made after seeing the data is a description, not a test. Concretely:
-
State the hypothesis and the number it predicts (pass-rate delta, activation score, firing rate) in the session log or spike note.
-
Validate the setup with zero spend first:
uv run python scripts/eval/eval-prompt-change.py \ --prompt templates/agents/analyst.shared.md \ --scenarios tests/evals/analyst-scenarios.json \ --base-ref main --dry-run(Both paths exist in-tree; there is no per-skill scenario fixture for every prompt. Substitute your own prompt path and a real
tests/evals/*.jsonfixture;ls tests/evals/lists what is available.)--dry-runvalidates inputs and makes no API calls (scripts/eval/eval-prompt-change.py:633,Validate inputs, no API calls). It is the only no-spend path; there is no--mock. -
Run the real eval, compare against the written prediction, and record both in the write-up. A miss is a finding, not an embarrassment.
Available instruments (all under scripts/eval/, verified 2026-07-03):
eval-prompt-change.py (before/after scenario judgment per ADR-057),
eval-agent-vs-baseline.py (agent prompt vs fixed baseline),
eval-rule-activation.py (does a .claude/rules/*.md file actually change
behavior: baseline vs description-only vs full-body, judged on activation,
citation, behavior). Interpretation guidance lives in
ai-agents-diagnostics-toolkit.
Process
The lifecycle. Every stage produces an artifact; every transition has a gate. Skipping a stage is how #1989 happened.
Phase 1: Hunch, Then Search, Then Contradiction Log
Before building anything, search: this codebase, Serena memories
(memory-search), ADRs, then external docs
(the programming-advisor skill's references/search-before-building.md). Most
hunches die here because the answer already exists; that is a cheap success.
If your first-principles position contradicts the conventional answer (an ADR,
a memory, a canonical pattern), log it before proceeding: Serena memory named
decision-<short-slug> with five fields: question,