跳至正文
OpenAgentSkill
浏览已开放 · 目录记录日期 2026-11-21 · 自动更新和账户服务暂时关闭。
开发与测试

prompt-regression

Use when the user has changed a prompt (system prompt, RAG template, agent instruction, etc.) and wants to know whether the candidate is better or worse than the baseline. Also use when the user mentions prompt A/B testing, prompt comparison, prompt optimization validation, "did my prompt change help," or prompt regression testing. Outputs per-dimension win rates with statistical significance using OpenJudge PairwiseAnalyzer.

agentscope-ai · 已有 AI 审核记录

安装与使用

运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。

npx skills add agentscope-ai/OpenJudge --skill prompt-regression

技能说明


name: prompt-regression description: > Use when the user has changed a prompt (system prompt, RAG template, agent instruction, etc.) and wants to know whether the candidate is better or worse than the baseline. Also use when the user mentions prompt A/B testing, prompt comparison, prompt optimization validation, "did my prompt change help," or prompt regression testing. Outputs per-dimension win rates with statistical significance using OpenJudge PairwiseAnalyzer.

Prompt Regression

Compare two prompts head-to-head and determine, with statistical rigor, whether the candidate is better, worse, or tied on each evaluation dimension.

When to Activate

  • You changed the system prompt and want to verify it's actually better
  • You're iterating on RAG answer templates
  • You're optimizing agent step-by-step instructions
  • You want data to support a prompt change decision

Checklist

You MUST create a task for each item and complete them in order:

  1. Load and analyze prompts — diff the baseline vs candidate
  2. Derive comparison dimensions — from the prompt changes + task type
  3. Select graders per dimension — pairwise, judge, or rule
  4. Run position-debiased comparison — swap-aggregate to eliminate order bias
  5. Compute statistics — win rates + bootstrap 95% CI per dimension
  6. Present results — per-dimension verdict with confidence intervals

Fast path: run the bundled script

Don't hand-write the win-rate + bootstrap math (the swap-aggregation and CI are easy to get wrong). Run the bundled, tested script (scripts/pairwise.py, standard library only, no OpenJudge dependency):

python scripts/pairwise.py --comparisons comparisons.jsonl --candidate candidate --baseline baseline

Each comparison row: {"id","model_a","model_b","score","dimension"?} where score >= 0.5 means model_a won. Emit two rows per query with A/B swapped to debias position. The script reports per-dimension candidate/baseline/tie rates, bootstrap 95% CI, and a verdict (BETTER / WORSE / TIED / INSUFFICIENT_EVIDENCE / INCONCLUSIVE; exit 0 only if better). --self-test to verify it.

Steps below explain how to derive dimensions and produce the comparisons (with OpenJudge or any judge); the inline snippets are the reference behind the script.

Step 1: Load and Analyze Prompts

Read the baseline and candidate prompts. Identify:

  • Task type: chatbot / RAG generation / code review / translation / summarization / agent instruction / other
  • What changed: added constraints, changed tone, new examples, different output format, expanded/shortened instructions
  • Intent of change: what problem was the user trying to fix?

Step 2: Derive Comparison Dimensions

Based on the task type and what changed, derive 3-5 comparison dimensions.

Dimension templates by task type

Chatbot / Conversational:

  • Answer relevance — does it address the user's question?
  • Tone appropriateness — does the tone match context?
  • Factual accuracy — no fabricated information
  • Conciseness — doesn't ramble or over-explain
  • Instruction following — obeys system prompt constraints

RAG Generation:

  • Faithfulness — grounded in retrieved documents
  • Citation accuracy — correctly references sources
  • Completeness — covers all aspects of the query
  • No hallucination — no claims beyond documents

Code Review / Generation:

  • Bug detection — finds real issues
  • False positive rate — doesn't flag correct code
  • Actionability — suggestions are specific and implementable
  • Code style — follows conventions

Agent Instructions:

  • Tool selection — picks the right tool
  • Step efficiency — minimal steps to goal
  • Error recovery — handles failures gracefully
  • Output format — follows specified structure

Each dimension gets:

  • An id (slug)
  • A one-sentence description
  • A grader type: pairwise or judge or rule

Step 3: Select Graders

Decision priority:

  1. Can a rule check this? → FunctionGrader or StringMatchGrader. Free, deterministic. Example: output length, keyword presence, JSON validity.
  2. Is there a reference answer? → pairwise against reference.
  3. Subjective quality, no reference? → pairwise A/B comparison.
  4. Single-output judgment needed? → judge (binary pass/fail per output).

Step 4: Run Position-Debiased Comparison

Pairwise comparison with swap-aggregate

LLM judges have position bias — the first response shown wins 5-15% more often. Swap-aggregate eliminates this: run each comparison twice with swapped positions, keep only consistent wins:

from openjudge.graders.llm_grader import LLMGrader
from openjudge.graders.schema import GraderMode
from openjudge.runner.grading_runner import GradingRunner
from openjudge.analyzer.pairwise_analyzer import PairwiseAnalyzer

# Judge prompt for relevance comparison
relevance_judge = LLMGrader(
    model=model,
    name="relevance_compare",
    mode=GraderMode.POINTWISE,
    template="""
Compare Response A and Response B for the query below.
Which response better addresses the user's question?

Query: {query}
Response A: {response_a}
Response B: {response_b}

Score 1.0 if A is better, 0.0 if B is better, 0.5 if tied.
Respond in JSON: {{"score": <float>, "reason": "<explanation>"}}
""",
)

# Build pairwise dataset with position swap
dataset = []
for sample in test_samples:
    # Original order
    dataset.append({
        "query": sample["query"],
        "response_a": baseline_outputs[sample["id"]],
        "response_b": candidate_outputs[sample["id"]],
        "metadata": {"model_a": "baseline", "model_b": "candidate"},
    })
    # Swapped order — critical for debiasing
    
阅读完整来源 ↗