跳至正文
OpenAgentSkill
浏览已开放 · 目录记录日期 2026-11-21 · 自动更新和账户服务暂时关闭。
搜索与研究

night-market-research-methodology

Turn hunches into accepted results: worthiness score, evidence bar, research-to-rules. Use when vetting ideas. Not for QA; use night-market-validation-and-qa.

athola · 已有 AI 审核记录

安装与使用

运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。

npx skills add athola/claude-night-market --skill night-market-research-methodology

技能说明


name: night-market-research-methodology description: "Turn hunches into accepted results: worthiness score, evidence bar, research-to-rules. Use when vetting ideas. Not for QA; use night-market-validation-and-qa."

Night Market Research Methodology

The discipline that turns a hunch into an accepted result in this repo. An "accepted result" is a change that survived the evidence bar and landed through change control as a rule, a skill module, a config gate, or an ADR. Everything else is either a local working note or a documented retirement. This skill covers the full path: score the idea, experiment behind a default-off flag, meet the evidence bar, land the durable artifact, or retire the idea on the record.

The evidence bar

A claim graduates from hunch to result only when it passes all four tests.

  1. One mechanism explains all observations, including negatives. If the hypothesis explains the three failing cases but not why the fourth case passed, it is incomplete. Keep digging until a single mechanism accounts for everything you saw.

  2. Predict numbers before running. Write down the expected measurement first, then measure. In-repo anchor: the forced-eval harness labels expected activations in prototypes/forced-eval/activation_cases.json before any run, then compares baseline against treatment with a McNemar paired test (a significance test for paired binary outcomes).

  3. Survive assigned adversarial refutation. Assign a reviewer or agent whose explicit job is to break the claim. Use Skill(attune:war-room) for hard-to-reverse decisions and Skill(imbue:rigorous-reasoning) to counter agreement bias. A claim nobody tried to break is unproven.

  4. Never let the generator judge itself. The agent that produced the work must not be its sole verifier. See plugins/imbue/skills/proof-of-work/modules/independent-verification.md. Prefer executable checks over an LLM judge, and prove the check can fail before trusting it (Guards 2 and 3 in plugins/imbue/skills/proof-of-work/modules/verifier-integrity.md).

Corollary from verifier-integrity: a green check proves the code satisfies the spec as written. It cannot prove the spec says what you meant, and it proves nothing if the check cannot fail. Validate the spec separately from the code, and mutation-test the check itself.

Idea lifecycle

An idea moves through four gates in order. Skipping a gate is how speculative infrastructure gets built and reverted.

Gate 1: score worthiness before building

Formula and thresholds from docs/backlog/queue.md (a local, gitignored working file):

Worthiness = (Business Value + Time Criticality + Risk Reduction)
           / (Complexity + Token Cost + Scope Drift)
ScoreAction
> 2.0Implement now
1.0 to 2.0Discuss before proceeding
< 1.0Keep in backlog

Queue rules: at most 10 active items. Items untouched for 30 days are archived to a GitHub issue (labels backlog,deferred) and removed from the queue. Because docs/backlog/ and docs/research/ are gitignored, the durable record of a deferred idea is the issue, not the queue file.

Gate 2: experiment behind a default-off flag

Exemplar: the egregore completion-integrity gate.

  • Commit 83281337 added the gate with completion_integrity: bool = False in plugins/egregore/scripts/config.py (still False as of 2026-07-02).
  • Commit cd903cbf added a test covering the raw-JSON opt-in path.

Pattern: land the mechanism off by default, cover the opt-in path with a test, and collect usage before proposing a default change.

Gate 3: data-collection window before structural change

ADR-0015 (docs/adr/0015-orchestrator-skill-simplification.md) requires 30 days of usage data before simplifying the over-built orchestrator skills. Apply the same bar to any promotion or simplification: name the data window in the PR, not an intuition.

Gate 4: adopt through change control or retire on the record

Adoption goes through the process in night-market-change-control. Retirement is written down, never silent. ADR-0012 (confidence-tagged claims) and ADR-0013 (Naur theory-building) carry Status: Superseded by ADR-0017, which is Accepted and rules "Do not build an enforcement mechanism. Permit voluntary use." A documented no is a valid result.

The research-to-rules pipeline

  1. Run multi-channel research (Skill(tome:research) or manual) into a dated synthesis at docs/research/YYYY-MM-DD-<topic>.md. Match the shape of the existing docs: Thesis, What the evidence says, solution pattern, Mapping to the night-market ecosystem, Evidence gaps and caveats.

  2. Map every gap against existing ecosystem assets before proposing new code. Most gaps turn out to be covered already (see case study 3).

  3. Land each real gap as the smallest durable artifact: a .claude/rules/ file, a module inside an existing skill, or a config gate. A new skill is the last resort (.claude/rules/shared-utility-consumer-rule.md requires 2+ consumers within 30 days).

  4. Fold the load-bearing evidence out of the synthesis and into the artifact that relies on it, as a table of sources and findings with resolvable identifiers (arXiv IDs, URLs), plus the caveats that bound them. Then delete the citation to the research file.

Caution: docs/research/ and docs/superpowers/ are both gitignored, so a tracked doc that cites a path under either is a dangling reference for every checkout but the author's. Step 4 is what prevents this, and it is not optional. Five syntheses were folded back into their consumers on 2026-07-27 for exactly this reason, and a brainstorm design record cited by .claude/rules/ceremony-requires-need.md survived that pass because the check below only looked at docs/research/. Verify with:

rg -o --hidden 'docs/(research|superpowers)/[A-Za-z0-9._/-]+\.md' \
   -g 
阅读完整来源 ↗