# Caliper: Know if your agent skill actually works

[](https://pypi.org/project/caliper-eval/) [](https://pypi.org/project/caliper-eval/) [](https://skills.sh/edonadei/caliper)

Caliper is a lightweight evaluation harness for agent skills. Write a short spec of what "good" looks like, run it again and again, and get a **success rate** you can track. It also runs every task _without_ the skill, so you learn whether the skill is doing the work or the base agent would have passed anyway. Works with the agent you already use: Claude Code, Codex, Pi, or Hermes.

**Teach your agent to evaluate:**

```bash npx skills@latest add edonadei/caliper ```

**Or run it yourself:**

```bash caliper run commit-commands.eval.yaml --k 3 --baseline ```

You write a spec, a few lines of YAML describing what "working" means, which you hand-write or have `/grill-skill` generate for you. With `--baseline`, Caliper runs each task with and without the skill and diffs the two runs task by task:

<!-- Terminal output of `caliper compare`, rendered to SVG so the box-drawing table stays aligned on every screen. Regenerate with: python docs/render_readme_samples.py --> 

---

Agent skills are hard to test. A skill that works on your machine, on this prompt, today, might fail tomorrow after a model update or a one-line prompt edit. Caliper makes reliability measurable: define what success looks like, run the skill repeatedly, and get a success rate you can track over time.

Use Caliper to answer questions like:

- Did my prompt edit actually improve the skill? - Is the skill doing the work, or would the base agent pass without it? - Does it still pa