跳至正文
OpenAgentSkill
浏览已开放 · 目录记录日期 2026-11-21 · 自动更新和账户服务暂时关闭。
设计与 UI

night-market-validation-and-qa

Enforce evidence bar, coverage gates, and regression guards. Use when adding tests or claiming done. Do not use to run suites; use night-market-operations.

athola · 已有 AI 审核记录

安装与使用

运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。

npx skills add athola/claude-night-market --skill night-market-validation-and-qa

技能说明


name: night-market-validation-and-qa description: 'Enforce evidence bar, coverage gates, and regression guards. Use when adding tests or claiming done. Do not use to run suites; use night-market-operations.'

Validation and QA: what counts as evidence here

This skill defines the evidence bar for claiming work complete, the coverage and quality thresholds that gate merges, the golden regression tests that guard settled battles, and the procedure for adding tests. The one-line thesis: a green check proves spec-satisfaction, not correctness, so every completion claim needs cited evidence and every test needs proof it can fail.

The evidence bar

Never claim "should work". Run the thing, capture the output, cite it. The house discipline comes from Skill(imbue:proof-of-work):

  1. Number every piece of evidence: [E1], [E2], each with the exact command and its captured output.
  2. Map each acceptance criterion to evidence with a verdict: Criterion: [E2] -> PASS or -> FAIL.
  3. Give the overall claim one of three statuses: COMPLETE (all criteria passed), PARTIAL (list blockers), BLOCKED (explain why). A blocked task reported as blocked with evidence is a successful report. A guessed "done" is not.
  4. The final response must not contain "should work", "looks right", or any other unverified confidence phrase.

Verifier integrity: never let the generator judge itself

A passing verifier can mislead two ways (from the prover-verifier research, codified in commit 29081fda, module plugins/imbue/skills/proof-of-work/modules/verifier-integrity.md):

Failure modeWhat it looks like
Wrong specThe check confirms the code matches the spec, not that the spec matches intent
Hollow checkassert True, a mock returning the expected value, a stubbed service: all green, all worthless

Rules that follow:

  • The agent that wrote the code must not be the sole judge of whether the code works. Use an independent check: a fresh subagent, the real test suite, a human, or an end-to-end run the generator cannot influence.
  • Validate the spec separately from the code. A machine-checked pass against a wrong spec is confident, green, and wrong.
  • When reviewing a test, ask what change to the code would make it fail. If nothing would, it is not a test.

Iron Law TDD (Constitution rule 3)

CONSTITUTION.md rule 3: no implementation without a failing test first. Scope:

  • Applies to every code change in plugin Python sources (plugins/*/src/, plugins/*/scripts/, plugins/*/hooks/).
  • Skill files, agent files, and prose docs are exempt. Their analogue is a structural validation test: every new skill needs a test_skill_<name>.py proving the structure.

A structural validation test asserts observable content in the skill file: required sections exist, the referenced modules exist on disk, tables and examples the skill promises are present. Model on plugins/imbue/tests/unit/skills/test_proof_of_work.py, which checks SKILL.md sections, module files, and enforcement tables. The test must fail if someone deletes the section it guards (see the tautological-test trap below).

Thresholds

GateValueWhere definedEnforced by
Root coveragefail_under = 85root pyproject.toml [tool.coverage.report]root pytest runs
Per-plugin coveragecoverage_threshold in [tool.nightmarket], 90 for most plugins, 85 for gauntletplugins/<p>/pyproject.tomlscripts/run-plugin-tests.sh passes --cov-fail-under only when the key is set and > 0
Mutation testingweekly Sunday 00:00 UTC cron mutates only sanctum (the matrix falls back to sanctum when the dispatch input is empty); abstract/imbue/attune run only via manual dispatch with plugin=all or a named plugin.github/workflows/mutation-testing.ymlmutmut: exit 0 = no survivors, exit 2 = survivors (allowed), anything else = crash
Critical issuesmax_critical_issues: 3, enforce_blocking: true.claude/quality_gates.jsonquality-gate tooling
File size< 20KB and < 5000 tokens per file.claude/quality_gates.jsonadvisory (block_on_violation: false)
Function length<= 60 lines, complexity < 12, nesting <= 5, debt ratio < 0.3.claude/quality_gates.jsonadvisory, except security dimension which blocks

Notes:

  • The run-plugin-tests.sh awk parser reads coverage_threshold from [tool.nightmarket], never from addopts. Use scripts/fix_coverage_threshold.py to migrate a plugin still using the old location.
  • The script itself sets no minimum floor for coverage_threshold. If the key is absent, no --cov-fail-under flag is passed at all, so a plugin without the key has no coverage gate in the runner.
  • imbue's own addopts force --cov=scripts plus term and HTML coverage reports on every run, including single-file runs.

The tautological-test trap

PR review here repeatedly catches tests that assert nothing. Evidence in history: a94240e2 (12 tests of constants tightened to behavioral coverage), f1cbbcf1 (strengthened tautological assertions), 30e58586 (validation-floor regression test), 42f7ce84 (round-trip test replacing a structure-only check). The pattern: a test that restates the code, mocks the unit under test, or asserts a constant equals itself.

The counter-discipline is the revert test, executed by sanctum:validate-pr:

  1. Take the fix the PR claims to make.
  2. Edit the fixed line back to its broken state (working tree must be clean first, or the step is skipped as unsafe).
  3. Run the test that supposedly guards the fix.
  4. The test must FAIL against the reverted code. If it stays green, the test is a dead assertion, not a guard.
  5. Restore the fix.

When writing a new test, apply the same standard preemptively: write the test so it would catch the bug's return, then confirm it fails be

阅读完整来源 ↗