Caliper
Lightweight evaluation harness for agent skills to track success rates across multiple agent runtimes.
edonadei · AI review on file
Installation
Read the source and dependencies before running a command. Copying does not install or execute anything.
npx skills add edonadei/caliperAbout this skill
Caliper: Know if your agent skill actually works
Caliper is a lightweight evaluation harness for agent skills. Write a short spec of what "good" looks like, run it again and again, and get a success rate you can track. It also runs every task without the skill, so you learn whether the skill is doing the work or the base agent would have passed anyway. Works with the agent you already use: Claude Code, Codex, Pi, or Hermes.
Teach your agent to evaluate:
npx skills@latest add edonadei/caliper
Or run it yourself:
caliper run commit-commands.eval.yaml --k 3 --baseline
You write a spec, a few lines of YAML describing what "working" means, which you hand-write or have /grill-skill generate for you. With --baseline, Caliper runs each task with and without the skill and diffs the two runs task by task:
Agent skills are hard to test. A skill that works on your machine, on this prompt, today, might fail tomorrow after a model update or a one-line prompt edit. Caliper makes reliability measurable: define what success looks like, run the skill repeatedly, and get a success rate you can track over time.
Use Caliper to answer questions like:
- Did my prompt edit actually improve the skill?
- Is the skill doing the work, or would the base agent pass without it?
- Does it still pa