Skip to content
OpenAgentSkill
Browsing is open · Catalog captured 2026-11-21 · Automatic updates and account services are paused.
Design & UI

ln-34-benchmark-comparator

Compares tools or implementations through reproducible A/B workloads, correctness oracles, and controlled measurements. Use to choose alternatives; not to optimize a known bottleneck.

levnikolaevich · AI review on file

Installation

Read the source and dependencies before running a command. Copying does not install or execute anything.

npx skills add levnikolaevich/claude-code-skills --skill ln-34-benchmark-comparator

About this skill


name: ln-34-benchmark-comparator description: "Compares tools or implementations through reproducible A/B workloads, correctness oracles, and controlled measurements. Use to choose alternatives; not to optimize a known bottleneck."

Benchmark Comparator

Goal: Compare alternatives under controlled, reproducible conditions. Correctness comes before speed, and measured data must remain separate from estimates, setup cost, and interpretation.

Execution contract: Treat the ordered checkbox workflow below as this skill's Definition of Done. Track every checkbox as PENDING, then resolve it to PROVEN with concrete evidence, CLEARED with evidence that its conditional trigger is absent, or UNPROVEN; reading, mentioning, delegating, skipping, or tool failure is not proof. Before returning, resolve every PENDING, count only PROVEN and CLEARED items as complete, apply this skill's verdict, decision, and approval rules to every UNPROVEN, and prepend Checklist: X/Y completeIncomplete: None | section/item — reason; outcome impact; exact next action; list every UNPROVEN item.

Tool Routing

NeedPreferred toolUse it whenFallback
Canonical workload and oracleRepository fixtures, tests, expected diffs, schemas, or independently specified outcomesDefining what success means before either candidate runsCreate the smallest deterministic fixture that represents the decision
IsolationClean Git worktrees, temporary directories, controlled environment, fixed seeds, and resettable cachesPreventing one candidate or run from contaminating anotherSequential clean-room setup with verified cleanup
ExecutionThe same shell runner and wrapper for every candidateCapturing commands, exit status, stdout, stderr, timing, and artifacts consistentlyManual execution with an explicit reproducibility limitation
Activation proofLogs, traces, command records, process metadata, or candidate-specific artifactsVerifying the intended alternative actually ran and did not fall backTreat the run as invalid when activation cannot be proven
Correctness gradingTests, output parser, diff, schema validation, or independent oracleEvery scenario before cost comparisonManual blind grading against written expectations
Performance and costMonotonic timer, resource metrics, token or usage telemetry, tool-call logs, and failure countsMetrics are observable through the same method for all candidatesLabel derived or estimated values and keep them out of measured aggregates
External semanticsOfficial documentation and specificationsCandidate configuration or claimed behavior needs current verificationPrimary-source web research; otherwise mark the claim UNVERIFIED

Do not tune the scenario after observing a preferred candidate, mix measurements from different workloads, or present internal estimates as externally measured facts. Benchmarking may create temporary worktrees and artifacts but must not change the source baseline or unapproved external state.

Evidence Rules

  • Specify scenarios, oracle, metrics, exclusions, and decision rule before running candidates.
  • Hold all non-tested variables constant or record and analyze the confounder.
  • Correctness failure cannot be compensated by better speed, token use, or cost unless the decision explicitly allows degraded correctness.
  • Use repeated runs and report raw values, center, spread, failures, and outliers; never headline the best run.
  • Keep setup or indexing cost, steady-state cost, maintenance burden, and runtime cost separate.
  • Report measured, derived, estimated, and qualitative evidence in distinct fields.

Checklist

1. Define the Decision and Experiment

  • State the decision the benchmark must support, the candidates, intended users, representative workloads, and explicit non-goals.
  • Include ordinary cases where a simpler or built-in candidate could reasonably win as well as cases exercising each candidate's claimed advantage; do not construct a feature demo for one side.
  • Define scenario inputs, expected outcomes, correctness criteria, failure conditions, and an oracle independent of candidate self-report.
  • Define primary and secondary metrics, units, measurement point, acceptance threshold, allowed tradeoffs, and tie or inconclusive rules.
  • Identify variables that must remain fixed: repository revision, model, prompts, permissions, runtime, hardware, data, cache policy, and network conditions.
  • Predeclare a pilot-derived repetition count tied to the minimum meaningful effect and observed noise, or a bounded sequential stopping rule with fixed error tolerance and maximum runs.
  • Freeze and hash the scenario text, fixtures, expectations, runner, parser, and decision rule before the first candidate result is inspected.
  • Read repository instructions and inspect Git state before creating worktrees, temporary data, or runners.
  • Start a run-owned resource ledger with every created absolute path, worktree, process ID, cache, account, dataset, report, and temporary artifact; never register pre-existing resources or credentials as cleanup targets.
  • Require side-effect-free or idempotent scenarios and disposable accounts or datasets; define external-write authorization, cost and rate budgets, cleanup, and rollback evidence before execution.
  • Return BLOCKED if candidates do not solve the same task, correctness cannot be independently graded, external effects cannot be isolated, or the decision rule is being chosen after results.

2. Build a Symmetric Harness

  • Use the same runner, timeout, logging, environment construction, and artifact collection for every candidate.
  • Create clean worktrees or equivalent isolated copies from the same commit and verify identical starting state.
  • Inventory global and user-level instructions, hooks, plugins, setting
Read the complete source ↗