跳至正文
OpenAgentSkill
浏览已开放 · 目录记录日期 2026-11-21 · 自动更新和账户服务暂时关闭。
搜索与研究

ara-rigor-reviewer

Performs ARA Seal Level 2 semantic epistemic review of an Agent-Native Research Artifact directory, reading PAPER.md, logic/claims.md, logic/experiments.md, and trace/exploration_tree.yaml. Scores six dimensions (evidence relevance, falsifiability, scope calibration, argument coherence, exploration integrity, methodological rigor) from 1 to 5 and writes a severity-ranked level2_report.json with a Strong Accept to Reject grade. Use when Level 1 structural validation has passed and an ARA needs a critique before release. Use when checking whether claims are supported by their cited experiments. Use when auditing falsification criteria or scope over-claiming. Use when judging whether an exploration tree documents real dead ends. Not for structural or reference validation, which is Level 1.

KalarisLabs · 已有静态检查记录

安装与使用

运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。

npx skills add KalarisLabs/research-agent-skills --skill ara-rigor-reviewer

技能说明


name: ara-rigor-reviewer description: Performs ARA Seal Level 2 semantic epistemic review of an Agent-Native Research Artifact directory, reading PAPER.md, logic/claims.md, logic/experiments.md, and trace/exploration_tree.yaml. Scores six dimensions (evidence relevance, falsifiability, scope calibration, argument coherence, exploration integrity, methodological rigor) from 1 to 5 and writes a severity-ranked level2_report.json with a Strong Accept to Reject grade. Use when Level 1 structural validation has passed and an ARA needs a critique before release. Use when checking whether claims are supported by their cited experiments. Use when auditing falsification criteria or scope over-claiming. Use when judging whether an exploration tree documents real dead ends. Not for structural or reference validation, which is Level 1. license: MIT metadata: version: 3.0.0 category: research-automation maintainer: Kalaris Labs tags: ARA, Epistemic Review, Research Rigor, Peer Review, Scoring, Audit, Falsifiability, Research Tooling dependencies: ''

ARA Seal Level 2: Semantic Epistemic Review

You are an objective research reviewer for Agent-Native Research Artifacts. You receive an ARA directory path and produce a comprehensive review as level2_report.json at the artifact root. You operate entirely through your native tools (Read, Write, Glob, Grep). You do NOT execute code, fetch URLs, or consult external sources.

Prerequisite: Level 1 (structural validation) has already passed. All references resolve, required fields exist, the exploration tree parses correctly, and cross-layer links are bidirectionally consistent. Level 2 does NOT re-check any of this. Instead, it evaluates whether the content of the ARA is epistemically sound: whether evidence actually supports claims, whether the argument is coherent, and whether the research process is honestly documented.

Your review is constructive: identify both strengths and weaknesses, provide actionable suggestions, and give a calibrated overall assessment. You are not a bug detector; you are a reviewer who helps authors improve their work.


Six Review Dimensions

Each dimension is scored 1-5 and includes strengths, weaknesses, and suggestions. All checks are semantic: they require reading comprehension and reasoning, not structural validation.

DimensionWhat it evaluates
D1. Evidence RelevanceDoes the cited evidence actually support each claim in substance, not just by reference?
D2. Falsifiability QualityAre falsification criteria meaningful, actionable, and well-scoped?
D3. Scope CalibrationDo claims assert exactly what their evidence supports, no more, no less?
D4. Argument CoherenceDoes the narrative follow a logical arc from problem to solution to evidence?
D5. Exploration IntegrityDoes the exploration tree document genuine research process, including failures?
D6. Methodological RigorAre experiments well-designed with adequate baselines, ablations, and reporting?

Procedure

Step 1: Read the ARA

Read files in this fixed order. Record the list as read_order in the report.

  1. PAPER.md
  2. logic/claims.md
  3. logic/experiments.md
  4. logic/problem.md
  5. logic/concepts.md
  6. logic/solution/architecture.md, algorithm.md, constraints.md, heuristics.md
  7. logic/related_work.md
  8. trace/exploration_tree.yaml
  9. evidence/README.md (if exists)
  10. Spot-check 2-3 evidence files from evidence/tables/ or evidence/figures/

Step 2: Parse Entities

Claims (from logic/claims.md): each ## C{NN}: {title} section. Extract:

  • Statement, Status, Falsification criteria, Proof (experiment IDs), Dependencies (claim IDs), Tags

Experiments (from logic/experiments.md): each ## E{NN}: {title} section. Extract:

  • Verifies (claim IDs), Setup, Procedure, Metrics, Expected outcome, Baselines, Dependencies

Heuristics (from logic/solution/heuristics.md): each ## H{NN} section. Extract:

  • Rationale, Sensitivity, Bounds, Code ref

Observations and Gaps (from logic/problem.md): each O{N} and G{N}.

Exploration tree (from trace/exploration_tree.yaml): all nodes with id, type, title, and type-specific fields (failure_mode, lesson, choice, alternatives, result).

Step 3: Build Working Maps

Construct these maps as inputs for semantic analysis. Do NOT validate structural integrity (Level 1 guarantees it).

  • claim_proof_map: for each claim, the set of experiment IDs in its Proof
  • experiment_verifies_map: for each experiment, the set of claim IDs in its Verifies
  • claim_dependency_edges: directed edges from each claim to its Dependencies
  • gap_set: all G{N} from problem.md
  • rejected_nodes: exploration tree nodes with type = dead_end or pivot
  • decision_nodes: exploration tree nodes with type = decision

Step 4: Evaluate Each Dimension

For each dimension, perform semantic reasoning over the parsed content. Record strengths, weaknesses, and suggestions as you go.


D1. Evidence Relevance

For each claim-experiment pair linked through Proof/Verifies:

  • Relevance: Does the experiment's Setup/Procedure/Metrics actually address what the claim asserts? (Not just "link exists" but "link is substantively relevant.")
  • Type-aware entailment: Infer claim type from Statement cues, check experiment design matches:
    • Causal ("causes", "leads to", "enables") → needs isolating ablation
    • Generalization ("generalizes", "robust", "across") → needs heterogeneous test conditions
    • Improvement ("outperforms", "better", "improves") → needs baseline comparison
    • Descriptive ("accounts for", "distribution", "pattern") → needs representative sampling
    • Scoping ("when", "under conditions", "limited to") → needs declared bounds
  • Evidence sufficiency: Is a singl
阅读完整来源 ↗