Skip to content
OpenAgentSkill
Browsing is open · Catalog captured 2026-11-21 · Automatic updates and account services are paused.
Development

eval-driven-development

Building rigorous LLM evaluation pipelines. Use when developing AI features to ensure quality and prevent regressions across different model versions.

ishandutta2007 · Static checks on file

Installation

Read the source and dependencies before running a command. Copying does not install or execute anything.

npx skills add ishandutta2007/Awesome-Agent-Skills --skill eval-driven-development

About this skill


name: eval-driven-development description: Building rigorous LLM evaluation pipelines. Use when developing AI features to ensure quality and prevent regressions across different model versions.

Eval-Driven Development

Overview

Eval-Driven Development ensures that AI features behave deterministically and predictably by testing them against a golden dataset using automated evaluators.

When to Use

  • Building an AI-powered feature
  • Tuning prompts or changing underlying models
  • Implementing RAG pipelines

Process

  1. Curate Golden Dataset: Create diverse test cases including edge cases.
  2. Define Metrics: Choose appropriate evaluators (e.g., exact match, semantic similarity, LLM-as-a-judge).
  3. Run Pipeline: Execute the AI feature over the dataset and collect results.
  4. Analyze Failures: Inspect low-scoring examples and update prompts or logic.
  5. Establish Baseline: Set a minimum threshold for CI/CD checks.

Common Rationalizations

RationalizationWhy It Is Wrong
"Manual spot checks are enough."Spot checks miss regressions across prompts, model versions, and edge cases.
"We can add evals after launch."Without a baseline, you cannot tell whether a later prompt or model change improved behavior.
"The judge model says it is good."LLM judges need criteria, calibration examples, and failure review before they are trustworthy.

Red Flags

  • No golden dataset exists
  • Metrics are vague or not tied to user-visible quality
  • Low-scoring examples are ignored instead of inspected
  • The baseline threshold is chosen after seeing the desired result

Verification

Before finishing, confirm:

  • The golden dataset includes normal, edge, and known-failure cases
  • Metrics and evaluator prompts are committed or otherwise reproducible
  • The current model/prompt has a recorded baseline
  • Failure examples have been reviewed and categorized