--- name: promptfoo-evals description: > Write, refine, run, and QA promptfoo evaluation suites: promptfooconfig.yaml, prompts, providers, vars, tests, assertions, model-graded rubrics, transforms, datasets, exports, and CI gates. Use for non-redteam eval coverage, regression tests, or new eval matrices. Do not use for adversarial redteam plugin or strategy setup. ---

# Writing Promptfoo Evals

You produce maintainable promptfoo eval suites: clear test cases, deterministic assertions where possible, model-graded only when needed.

See `references/cheatsheet.md` for the full assertion and provider reference. For deep questions about promptfoo features, consult https://www.promptfoo.dev/llms-full.txt

## Inputs (infer from repo context if not provided)

- What is being evaluated (prompt, agent, endpoint, RAG pipeline)? - What are the inputs and outputs (text, JSON, multi-turn chat, tool calls)? - What does "good" look like (acceptance criteria, failure modes)?

If context is insufficient, scaffold with TODO markers and starter tests.

## Workflow

### 1. Find or create the eval suite

Search for existing configs: `promptfooconfig.yaml`, `promptfooconfig.yml`, or any `promptfoo`/`evals` folder. Extend existing suites when possible.

For new suites, use this layout (unless the repo uses another convention):

```text evals/<suite-name>/ promptfooconfig.yaml prompts/ tests/ ```

Always add `# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json` at the top of config files.

### 2. Write prompts

- Put prompts in `prompts/*.txt` (plain) or `prompts/*.json` (chat format) - Reference via `file://prompts/main.txt` - Use `{{variable}}` for test inputs - If the app builds prompts dynamically, use a JS/Python provider instead of duplicating logic

### 3. Choose providers

Pick the simplest option that matches the real system:

| Scenario | Provider pattern | | ---------------- | --------------------------------------------------------------------- | | Compare models | `openai:chat:gpt-4.1-mini`, `anthropic:messages:claude-sonnet-4-6` | | Test an HTTP API | `id: https` with `config.url`, `config.body`, and `transformResponse` | | Test local code | `file://provider.py` or `file://provider.js` | | Echo/passthrough | `echo` (returns prompt as-is, useful for testing assertions) |

Keep provider count small: 1 for regression, 2 for comparison.

For JSON output, add `response_format` to the provider config:

```yaml config: temperature: 0 response_format: type: json_object ```

### 4. Write tests

Use file-based tests so they scale: `tests: file://tests/*.yaml`

For larger suites, use dataset-backed tests:

```yaml tests: file://tests.csv # or tests: file://generate_tests.py:create_tests ```

Every test should have:

- `description` - short, specific - `vars` - the inputs - `assert` - validations (when automatable)

Cover: happy paths, edge cases, known regressions, safety/refusal checks, output format compliance.

### 5. Add assertions

**Deterministic first** (fast, reliable, free): `equals`, `contains`, `icontains`, `regex`, `is-json`, `contains-json`, `starts-with`, `cost`, `latency`, `javascript`, `python`

**Model-graded sparingly** (slow, costs money, non-deterministic): `llm-rubric`, `factuality`, `answer-relevance`, `context-faithfulness`

Assertions support optional `weight` (for scoring relative importance) and `metric` (named score in reports). `threshold` is assertion-specific: for graded assertions it is usually a minimum score (0-1), while for assertions like `cost`/`latency` it is a maximum allowed value.

For model-graded assertions, explicitly set the grader provider so grading is stable across runs:

```yaml defaultTest: options: provider: openai:gpt-5-mini

tests: - description: 'Model-graded quality check' assert: - type: llm-rubric value: 'Accurate and concise' # Optional per-assertion override: # provider: anthropic:messages:claude-sonnet-4-6 ```

**Hallucination / faithfulness pattern:** When checking that output is grounded in source material, include the source in the rubric so the grader can compare. Use `context-faithfulness` when you have a context var, or inline the source in the `llm-rubric` value:

```yaml assert: - type: llm-rubric value: | The summary only states facts from this source article: "{{article}}" It does not add, infer, or fabricate any claims. ```

**JSON output pattern:**

```yaml assert: - type: is-json value: # optional JSON Schema type: object required: [name, score] - type: javascript value: 'JSON.parse(output).score >= 0.8' ```

**Transform pattern** (preprocess output before assertions): When models wrap JSON in markdown fences or add preamble text, use `options.transform` on the test to clean output before assertions run:

````yaml options: transform: "output.replace(/```json\\n?|```/g, '').trim()" ````

Use `defaultTest` for assertions shared across all tests (cost limits, format checks, etc.).

### 6. Validate and run

Before finishing, validate and provide run commands. Always use `--no-cache` during development to avoid stale results. Only run eval if credentials are available and safe to call.

```bash npx promptfoo@latest validate config -c <config> npx promptfoo@latest eval -c <config> -o output.json --no-cache --no-share ```

For CI/non-UI workflows, prefer the `-o output.json` command and inspect `success`, `score`, and `error` fields.

If working in the promptfoo repo itself, prefer the local build:

```bash source ~/.nvm/nvm.sh && nvm use npm run local -- validate config -c <config> npm run local -- eval -c <config> -o output.json --no-cache --no-share ```

Add `--env-file .env` only when the eval needs local credentials and that file exists.

Do not run `npm run local -- view` unless explicitly asked.

## Common mistakes

```yaml # ❌ WRONG — shell-style env vars don't work in YAML configs apiKey: $OPENAI_API_KEY

# ✅ CORRECT — use Nunjucks syntax with quotes apiKey: '{{env.OPENAI_API_KEY}}' ```

```yaml # ❌ WRONG — rubric references "the article" but grader can't see it - type: llm-rubric value: 'Only contains info from the original article'

# ✅ CORRECT — inline the source so the grader can compare - type: llm-rubric value: | Only states facts from: "{{article}}" ```

## Output contract

When done, state:

- What the suite evaluates (1-3 bullets) - Files created/modified (paths) - How to run (copy-pastable commands) - Required env vars - TODOs left behind (only if unavoidable)

--- name: promptfoo-evals description: > Write, refine, run, and QA promptfoo evaluation suites: promptfooconfig.yaml, prompts, providers, vars, tests, assertions, model-graded rubrics, transforms, datasets, exports, and CI gates. Use for non-redteam eval coverage, regression tests, or new eval matrices. Do not use for adversarial redteam plugin or strategy setup. ---

# Writing Promptfoo Evals

You produce maintainable promptfoo eval suites: clear test cases, deterministic assertions where possible, model-graded only when needed.

See `references/cheatsheet.md` for the full assertion and provider reference. For deep questions about promptfoo features, consult https://www.promptfoo.dev/llms-full.txt

## Inputs (infer from repo context if not provided)

- What is being evaluated (prompt, agent, endpoint, RAG pipeline)? - What are the inputs and outputs (text, JSON, multi-turn chat, tool calls)? - What does "good" look like (acceptance criteria, failure modes)?

If context is insufficient, scaffold with TODO markers and starter tests.

## Workflow

### 1. Find or create the eval suite

Search for existing configs: `promptfooconfig.yaml`, `promptfooconfig.yml`, or any `promptfoo`/`evals` folder. Extend existing suites when possible.

For new suites, use this layout (unless the repo uses another convention):

```text evals/<suite-name>/ promptfooconfig.yaml prompts/ tests/ ```

Always add `# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json` at the top of config files.

### 2. Write prompts

- Put prompts in `prompts/*.txt` (plain) or `prompts/*.json` (chat format) - Reference via `file://prompts/main.txt` - Use `{{variable}}` for test inputs - If the app builds prompts dynamically, use a JS/Python provider instead of duplicating logic

### 3. Choose providers

Pick the simplest option that matches the real system:

| Scenario | Provider pattern | | ---------------- | --------------------------------------------------------------------- | | Compare models | `openai:chat:gpt-4.1-mini`, `anthropic:messages:claude-sonnet-4-6` | | Test an HTTP API | `id: https` with `config.url`, `config.body`, and `transformResponse` | | Test local code | `file://provider.py` or `file://provider.js` | | Echo/passthrough | `echo` (returns prompt as-is, useful for testing assertions) |

Keep provider count small: 1 for regression, 2 for comparison.

For JSON output, add `response_format` to the provider config:

```yaml config: temperature: 0 response_format: type: json_object ```

### 4. Write tests

Use file-based tests so they scale: `tests: file://tests/*.yaml`

For larger suites, use dataset-backed tests:

```yaml tests: file://tests.csv # or tests: file://generate_tests.py:create_tests ```

Every test should have:

- `description` - short, specific - `vars` - the inputs - `assert` - validations (when automatable)

Cover: happy paths, edge cases, known regressions, safety/refusal checks, output format compliance.

### 5. Add assertions

**Deterministic first** (fast, reliable, free): `equals`, `contains`, `icontains`, `regex`, `is-json`, `contains-json`, `starts-with`, `cost`, `latency`, `javascript`, `python`

**Model-graded sparingly** (slow, costs money, non-deterministic): `llm-rubric`, `factuality`, `answer-relevance`, `context-faithfulness`

Assertions support optional `weight` (for scoring relative importance) and `metric` (named score in reports). `threshold` is assertion-specific: for graded assertions it is usually a minimum score (0-1), while for assertions like `cost`/`latency` it is a maximum allowed value.

For model-graded assertions, explicitly set the grader provider so grading is stable across runs:

```yaml defaultTest: options: provider: openai:gpt-5-mini

tests: - description: 'Model-graded quality check' assert: - type: llm-rubric value: 'Accurate and concise' # Optional per-assertion override: # provider: anthropic:messages:claude-sonnet-4-6 ```

**Hallucination / faithfulness pattern:** When checking that output is grounded in source material, include the source in the rubric so the grader can compare. Use `context-faithfulness` when you have a context var, or inline the source in the `llm-rubric` value:

```yaml assert: - type: llm-rubric value: | The summary only states facts from this source article: "{{article}}" It does not add, infer, or fabricate any claims. ```

**JSON output pattern:**

```yaml assert: - type: is-json value: # optional JSON Schema type: object required: [name, score] - type: javascript value: 'JSON.parse(output).score >= 0.8' ```

**Transform pattern** (preprocess output before assertions): When models wrap JSON in markdown fences or add preamble text, use `options.transform` on the test to clean output before assertions run:

````yaml options: transform: "output.replace(/```json\\n?|```/g, '').trim()" ````

Use `defaultTest` for assertions shared across all tests (cost limits, format checks, etc.).

### 6. Validate and run

Before finishing, validate and provide run commands. Always use `--no-cache` during development to avoid stale results. Only run eval if credentials are available and safe to call.

```bash npx promptfoo@latest validate config -c <config> npx promptfoo@latest eval -c <config> -o output.json --no-cache --no-share ```

For CI/non-UI workflows, prefer the `-o output.json` command and inspect `success`, `score`, and `error` fields.

If working in the promptfoo repo itself, prefer the local build:

```bash source ~/.nvm/nvm.sh && nvm use npm run local -- validate config -c <config> npm run local -- eval -c <config> -o output.json --no-cache --no-share ```

Add `--env-file .env` only when the eval needs local credentials and that file exists.

Do not run `npm run local -- view` unless explicitly asked.

## Common mistakes

```yaml # ❌ WRONG — shell-style env vars don't work in YAML configs apiKey: $OPENAI_API_KEY

# ✅ CORRECT — use Nunjucks syntax with quotes apiKey: '{{env.OPENAI_API_KEY}}' ```

```yaml # ❌ WRONG — rubric references "the article" but grader can't see it - type: llm-rubric value: 'Only contains info from the original article'

# ✅ CORRECT — inline the source so the grader can compare - type: llm-rubric value: | Only states facts from: "{{article}}" ```

## Output contract

When done, state:

- What the suite evaluates (1-3 bullets) - Files created/modified (paths) - How to run (copy-pastable commands) - Required env vars - TODOs left behind (only if unavoidable)

--- name: promptfoo-evals description: > Write, refine, run, and QA promptfoo evaluation suites: promptfooconfig.yaml, prompts, providers, vars, tests, assertions, model-graded rubrics, transforms, datasets, exports, and CI gates. Use for non-redteam eval coverage, regression tests, or new eval matrices. Do not use for adversarial redteam plugin or strategy setup. ---

# Writing Promptfoo Evals

You produce maintainable promptfoo eval suites: clear test cases, deterministic assertions where possible, model-graded only when needed.

See `references/cheatsheet.md` for the full assertion and provider reference. For deep questions about promptfoo features, consult https://www.promptfoo.dev/llms-full.txt

## Inputs (infer from repo context if not provided)

- What is being evaluated (prompt, agent, endpoint, RAG pipeline)? - What are the inputs and outputs (text, JSON, multi-turn chat, tool calls)? - What does "good" look like (acceptance criteria, failure modes)?

If context is insufficient, scaffold with TODO markers and starter tests.

## Workflow

### 1. Find or create the eval suite

Search for existing configs: `promptfooconfig.yaml`, `promptfooconfig.yml`, or any `promptfoo`/`evals` folder. Extend existing suites when possible.

For new suites, use this layout (unless the repo uses another convention):

```text evals/<suite-name>/ promptfooconfig.yaml prompts/ tests/ ```

Always add `# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json` at the top of config files.

### 2. Write prompts

- Put prompts in `prompts/*.txt` (plain) or `prompts/*.json` (chat format) - Reference via `file://prompts/main.txt` - Use `{{variable}}` for test inputs - If the app builds prompts dynamically, use a JS/Python provider instead of duplicating logic

### 3. Choose providers

Pick the simplest option that matches the real system:

| Scenario | Provider pattern | | ---------------- | --------------------------------------------------------------------- | | Compare models | `openai:chat:gpt-4.1-mini`, `anthropic:messages:claude-sonnet-4-6` | | Test an HTTP API | `id: https` with `config.url`, `config.body`, and `transformResponse` | | Test local code | `file://provider.py` or `file://provider.js` | | Echo/passthrough | `echo` (returns prompt as-is, useful for testing assertions) |

Keep provider count small: 1 for regression, 2 for comparison.

For JSON output, add `response_format` to the provider config:

```yaml config: temperature: 0 response_format: type: json_object ```

### 4. Write tests

Use file-based tests so they scale: `tests: file://tests/*.yaml`

For larger suites, use dataset-backed tests:

```yaml tests: file://tests.csv # or tests: file://generate_tests.py:create_tests ```

Every test should have:

- `description` - short, specific - `vars` - the inputs - `assert` - validations (when automatable)

Cover: happy paths, edge cases, known regressions, safety/refusal checks, output format compliance.

### 5. Add assertions

**Deterministic first** (fast, reliable, free): `equals`, `contains`, `icontains`, `regex`, `is-json`, `contains-json`, `starts-with`, `cost`, `latency`, `javascript`, `python`

**Model-graded sparingly** (slow, costs money, non-deterministic): `llm-rubric`, `factuality`, `answer-relevance`, `context-faithfulness`

Assertions support optional `weight` (for scoring relative importance) and `metric` (named score in reports). `threshold` is assertion-specific: for graded assertions it is usually a minimum score (0-1), while for assertions like `cost`/`latency` it is a maximum allowed value.

For model-graded assertions, explicitly set the grader provider so grading is stable across runs:

```yaml defaultTest: options: provider: openai:gpt-5-mini

tests: - description: 'Model-graded quality check' assert: - type: llm-rubric value: 'Accurate and concise' # Optional per-assertion override: # provider: anthropic:messages:claude-sonnet-4-6 ```

**Hallucination / faithfulness pattern:** When checking that output is grounded in source material, include the source in the rubric so the grader can compare. Use `context-faithfulness` when you have a context var, or inline the source in the `llm-rubric` value:

```yaml assert: - type: llm-rubric value: | The summary only states facts from this source article: "{{article}}" It does not add, infer, or fabricate any claims. ```

**JSON output pattern:**

```yaml assert: - type: is-json value: # optional JSON Schema type: object required: [name, score] - type: javascript value: 'JSON.parse(output).score >= 0.8' ```

**Transform pattern** (preprocess output before assertions): When models wrap JSON in markdown fences or add preamble text, use `options.transform` on the test to clean output before assertions run:

````yaml options: transform: "output.replace(/```json\\n?|```/g, '').trim()" ````

Use `defaultTest` for assertions shared across all tests (cost limits, format checks, etc.).

### 6. Validate and run

Before finishing, validate and provide run commands. Always use `--no-cache` during development to avoid stale results. Only run eval if credentials are available and safe to call.

```bash npx promptfoo@latest validate config -c <config> npx promptfoo@latest eval -c <config> -o output.json --no-cache --no-share ```

For CI/non-UI workflows, prefer the `-o output.json` command and inspect `success`, `score`, and `error` fields.

If working in the promptfoo repo itself, prefer the local build:

```bash source ~/.nvm/nvm.sh && nvm use npm run local -- validate config -c <config> npm run local -- eval -c <config> -o output.json --no-cache --no-share ```

Add `--env-file .env` only when the eval needs local credentials and that file exists.

Do not run `npm run local -- view` unless explicitly asked.

## Common mistakes

```yaml # ❌ WRONG — shell-style env vars don't work in YAML configs apiKey: $OPENAI_API_KEY

# ✅ CORRECT — use Nunjucks syntax with quotes apiKey: '{{env.OPENAI_API_KEY}}' ```

```yaml # ❌ WRONG — rubric references "the article" but grader can't see it - type: llm-rubric value: 'Only contains info from the original article'

# ✅ CORRECT — inline the source so the grader can compare - type: llm-rubric value: | Only states facts from: "{{article}}" ```

## Output contract

When done, state:

- What the suite evaluates (1-3 bullets) - Files created/modified (paths) - How to run (copy-pastable commands) - Required env vars - TODOs left behind (only if unavoidable)

--- name: promptfoo-evals description: > Write, refine, run, and QA promptfoo evaluation suites: promptfooconfig.yaml, prompts, providers, vars, tests, assertions, model-graded rubrics, transforms, datasets, exports, and CI gates. Use for non-redteam eval coverage, regression tests, or new eval matrices. Do not use for adversarial redteam plugin or strategy setup. ---

# Writing Promptfoo Evals

You produce maintainable promptfoo eval suites: clear test cases, deterministic assertions where possible, model-graded only when needed.

See `references/cheatsheet.md` for the full assertion and provider reference. For deep questions about promptfoo features, consult https://www.promptfoo.dev/llms-full.txt

## Inputs (infer from repo context if not provided)

- What is being evaluated (prompt, agent, endpoint, RAG pipeline)? - What are the inputs and outputs (text, JSON, multi-turn chat, tool calls)? - What does "good" look like (acceptance criteria, failure modes)?

If context is insufficient, scaffold with TODO markers and starter tests.

## Workflow

### 1. Find or create the eval suite

Search for existing configs: `promptfooconfig.yaml`, `promptfooconfig.yml`, or any `promptfoo`/`evals` folder. Extend existing suites when possible.

For new suites, use this layout (unless the repo uses another convention):

```text evals/<suite-name>/ promptfooconfig.yaml prompts/ tests/ ```

Always add `# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json` at the top of config files.

### 2. Write prompts

- Put prompts in `prompts/*.txt` (plain) or `prompts/*.json` (chat format) - Reference via `file://prompts/main.txt` - Use `{{variable}}` for test inputs - If the app builds prompts dynamically, use a JS/Python provider instead of duplicating logic

### 3. Choose providers

Pick the simplest option that matches the real system:

| Scenario | Provider pattern | | ---------------- | --------------------------------------------------------------------- | | Compare models | `openai:chat:gpt-4.1-mini`, `anthropic:messages:claude-sonnet-4-6` | | Test an HTTP API | `id: https` with `config.url`, `config.body`, and `transformResponse` | | Test local code | `file://provider.py` or `file://provider.js` | | Echo/passthrough | `echo` (returns prompt as-is, useful for testing assertions) |

Keep provider count small: 1 for regression, 2 for comparison.

For JSON output, add `response_format` to the provider config:

```yaml config: temperature: 0 response_format: type: json_object ```

### 4. Write tests

Use file-based tests so they scale: `tests: file://tests/*.yaml`

For larger suites, use dataset-backed tests:

```yaml tests: file://tests.csv # or tests: file://generate_tests.py:create_tests ```

Every test should have:

- `description` - short, specific - `vars` - the inputs - `assert` - validations (when automatable)

Cover: happy paths, edge cases, known regressions, safety/refusal checks, output format compliance.

### 5. Add assertions

**Deterministic first** (fast, reliable, free): `equals`, `contains`, `icontains`, `regex`, `is-json`, `contains-json`, `starts-with`, `cost`, `latency`, `javascript`, `python`

**Model-graded sparingly** (slow, costs money, non-deterministic): `llm-rubric`, `factuality`, `answer-relevance`, `context-faithfulness`

Assertions support optional `weight` (for scoring relative importance) and `metric` (named score in reports). `threshold` is assertion-specific: for graded assertions it is usually a minimum score (0-1), while for assertions like `cost`/`latency` it is a maximum allowed value.

For model-graded assertions, explicitly set the grader provider so grading is stable across runs:

```yaml defaultTest: options: provider: openai:gpt-5-mini

tests: - description: 'Model-graded quality check' assert: - type: llm-rubric value: 'Accurate and concise' # Optional per-assertion override: # provider: anthropic:messages:claude-sonnet-4-6 ```

**Hallucination / faithfulness pattern:** When checking that output is grounded in source material, include the source in the rubric so the grader can compare. Use `context-faithfulness` when you have a context var, or inline the source in the `llm-rubric` value:

```yaml assert: - type: llm-rubric value: | The summary only states facts from this source article: "{{article}}" It does not add, infer, or fabricate any claims. ```

**JSON output pattern:**

```yaml assert: - type: is-json value: # optional JSON Schema type: object required: [name, score] - type: javascript value: 'JSON.parse(output).score >= 0.8' ```

**Transform pattern** (preprocess output before assertions): When models wrap JSON in markdown fences or add preamble text, use `options.transform` on the test to clean output before assertions run:

````yaml options: transform: "output.replace(/```json\\n?|```/g, '').trim()" ````

Use `defaultTest` for assertions shared across all tests (cost limits, format checks, etc.).

### 6. Validate and run

Before finishing, validate and provide run commands. Always use `--no-cache` during development to avoid stale results. Only run eval if credentials are available and safe to call.

```bash npx promptfoo@latest validate config -c <config> npx promptfoo@latest eval -c <config> -o output.json --no-cache --no-share ```

For CI/non-UI workflows, prefer the `-o output.json` command and inspect `success`, `score`, and `error` fields.

If working in the promptfoo repo itself, prefer the local build:

```bash source ~/.nvm/nvm.sh && nvm use npm run local -- validate config -c <config> npm run local -- eval -c <config> -o output.json --no-cache --no-share ```

Add `--env-file .env` only when the eval needs local credentials and that file exists.

Do not run `npm run local -- view` unless explicitly asked.

## Common mistakes

```yaml # ❌ WRONG — shell-style env vars don't work in YAML configs apiKey: $OPENAI_API_KEY

# ✅ CORRECT — use Nunjucks syntax with quotes apiKey: '{{env.OPENAI_API_KEY}}' ```

```yaml # ❌ WRONG — rubric references "the article" but grader can't see it - type: llm-rubric value: 'Only contains info from the original article'

# ✅ CORRECT — inline the source so the grader can compare - type: llm-rubric value: | Only states facts from: "{{article}}" ```

## Output contract

When done, state:

- What the suite evaluates (1-3 bullets) - Files created/modified (paths) - How to run (copy-pastable commands) - Required env vars - TODOs left behind (only if unavoidable)