# cultivar

A CLI tool to help you write tests for skills, test them across agents, and iterate until they work, from the Pinecone DevRel team.

Test how well skills work against tasks, across agents, locally and remotely. Customize sandboxes for how agents should start, and graders for how agents should work.

Use traces to iteratively refine skills and optimize them against tasks.

Benchmark against skills, docs, and baselines. And, even run in parallel simulatenously for faster execution.

**Two ways to run it:** the `cultivar` CLI directly, or install the bundled skill and let your coding agent drive it:

```bash npx skills add https://github.com/pinecone-io/cultivar --skill cultivar ```

Same engine either way — but when an agent drives it you won't watch the live run / `--remote` sandbox dashboard as directly as in your own terminal. Keep the skill project-scoped (not `-g`); it's never auto-tested.

## Prerequisites

- Python 3.11+ and [uv](https://docs.astral.sh/uv/) - An Anthropic API key (the grader runs locally; agents in the sandbox use Modal-injected keys) - A [Modal](https://modal.com) account if you want `--remote` runs (recommended for parallelism + isolation).**This is the recommended experience!**

## Install

```bash uv tool install cultivar

# Or install from source: uv tool install --from "git+https://github.com/pinecone-io/cultivar" cultivar ```

## Modal setup (one-time)

`--remote` runs each eval in an isolated [Modal](https://modal.com) sandbox — recommended for parallelism and clean auth state. Skip this section if you only need local runs.

```bash # 1. Install Modal and authenticate pip install modal modal token new

# 2. Create the secret the sandbox reads at runtime modal secret create eval-sandbox-secrets \ ANTHROPIC_API_KEY=sk-ant-... # Add any keys your tasks need: GEMINI_API_KEY, COPILOT_GITHUB_TOKEN, etc.

# 3. Verify modal secret list # eval-sandbox-secrets should appear ```

The first `--remote` run builds the sandbox imag