Skip to content
OpenAgentSkill
Browsing is open · Catalog captured 2026-11-21 · Automatic updates and account services are paused.
Development

Caliper

Lightweight evaluation harness for agent skills to track success rates across multiple agent runtimes.

edonadei · AI review on file

Installation

Read the source and dependencies before running a command. Copying does not install or execute anything.

npx skills add edonadei/caliper

About this skill

Caliper: Know if your agent skill actually works

Caliper is a lightweight evaluation harness for agent skills. Write a short spec of what "good" looks like, run it again and again, and get a success rate you can track. It also runs every task without the skill, so you learn whether the skill is doing the work or the base agent would have passed anyway. Works with the agent you already use: Claude Code, Codex, Pi, or Hermes.

Teach your agent to evaluate:

npx skills@latest add edonadei/caliper

Or run it yourself:

caliper run commit-commands.eval.yaml --k 3 --baseline

You write a spec, a few lines of YAML describing what "working" means, which you hand-write or have /grill-skill generate for you. With --baseline, Caliper runs each task with and without the skill and diffs the two runs task by task:


Agent skills are hard to test. A skill that works on your machine, on this prompt, today, might fail tomorrow after a model update or a one-line prompt edit. Caliper makes reliability measurable: define what success looks like, run the skill repeatedly, and get a success rate you can track over time.

Use Caliper to answer questions like:

  • Did my prompt edit actually improve the skill?
  • Is the skill doing the work, or would the base agent pass without it?
  • Does it still pa