prompt-regression
Use when the user has changed a prompt (system prompt, RAG template, agent instruction, etc.) and wants to know whether the candidate is better or worse than the baseline. Also use when the user mentions prompt A/B testing, prompt comparison, prompt optimization validation, "did my prompt change help," or prompt regression testing. Outputs per-dimension win rates with statistical significance using OpenJudge PairwiseAnalyzer.
agentscope-ai · 已有 AI 审核记录
安装与使用
运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。
npx skills add agentscope-ai/OpenJudge --skill prompt-regression技能说明
name: prompt-regression description: > Use when the user has changed a prompt (system prompt, RAG template, agent instruction, etc.) and wants to know whether the candidate is better or worse than the baseline. Also use when the user mentions prompt A/B testing, prompt comparison, prompt optimization validation, "did my prompt change help," or prompt regression testing. Outputs per-dimension win rates with statistical significance using OpenJudge PairwiseAnalyzer.
Prompt Regression
Compare two prompts head-to-head and determine, with statistical rigor, whether the candidate is better, worse, or tied on each evaluation dimension.
When to Activate
- You changed the system prompt and want to verify it's actually better
- You're iterating on RAG answer templates
- You're optimizing agent step-by-step instructions
- You want data to support a prompt change decision
Checklist
You MUST create a task for each item and complete them in order:
- Load and analyze prompts — diff the baseline vs candidate
- Derive comparison dimensions — from the prompt changes + task type
- Select graders per dimension — pairwise, judge, or rule
- Run position-debiased comparison — swap-aggregate to eliminate order bias
- Compute statistics — win rates + bootstrap 95% CI per dimension
- Present results — per-dimension verdict with confidence intervals
Fast path: run the bundled script
Don't hand-write the win-rate + bootstrap math (the swap-aggregation and CI are easy to get
wrong). Run the bundled, tested script (scripts/pairwise.py, standard library only, no
OpenJudge dependency):
python scripts/pairwise.py --comparisons comparisons.jsonl --candidate candidate --baseline baseline
Each comparison row: {"id","model_a","model_b","score","dimension"?} where score >= 0.5
means model_a won. Emit two rows per query with A/B swapped to debias position. The
script reports per-dimension candidate/baseline/tie rates, bootstrap 95% CI, and a verdict
(BETTER / WORSE / TIED / INSUFFICIENT_EVIDENCE / INCONCLUSIVE; exit 0 only if
better). --self-test to verify it.
Steps below explain how to derive dimensions and produce the comparisons (with OpenJudge or any judge); the inline snippets are the reference behind the script.
Step 1: Load and Analyze Prompts
Read the baseline and candidate prompts. Identify:
- Task type: chatbot / RAG generation / code review / translation / summarization / agent instruction / other
- What changed: added constraints, changed tone, new examples, different output format, expanded/shortened instructions
- Intent of change: what problem was the user trying to fix?
Step 2: Derive Comparison Dimensions
Based on the task type and what changed, derive 3-5 comparison dimensions.
Dimension templates by task type
Chatbot / Conversational:
- Answer relevance — does it address the user's question?
- Tone appropriateness — does the tone match context?
- Factual accuracy — no fabricated information
- Conciseness — doesn't ramble or over-explain
- Instruction following — obeys system prompt constraints
RAG Generation:
- Faithfulness — grounded in retrieved documents
- Citation accuracy — correctly references sources
- Completeness — covers all aspects of the query
- No hallucination — no claims beyond documents
Code Review / Generation:
- Bug detection — finds real issues
- False positive rate — doesn't flag correct code
- Actionability — suggestions are specific and implementable
- Code style — follows conventions
Agent Instructions:
- Tool selection — picks the right tool
- Step efficiency — minimal steps to goal
- Error recovery — handles failures gracefully
- Output format — follows specified structure
Each dimension gets:
- An
id(slug) - A one-sentence description
- A grader type:
pairwiseorjudgeorrule
Step 3: Select Graders
Decision priority:
- Can a rule check this? →
FunctionGraderorStringMatchGrader. Free, deterministic. Example: output length, keyword presence, JSON validity. - Is there a reference answer? →
pairwiseagainst reference. - Subjective quality, no reference? →
pairwiseA/B comparison. - Single-output judgment needed? →
judge(binary pass/fail per output).
Step 4: Run Position-Debiased Comparison
Pairwise comparison with swap-aggregate
LLM judges have position bias — the first response shown wins 5-15% more often. Swap-aggregate eliminates this: run each comparison twice with swapped positions, keep only consistent wins:
from openjudge.graders.llm_grader import LLMGrader
from openjudge.graders.schema import GraderMode
from openjudge.runner.grading_runner import GradingRunner
from openjudge.analyzer.pairwise_analyzer import PairwiseAnalyzer
# Judge prompt for relevance comparison
relevance_judge = LLMGrader(
model=model,
name="relevance_compare",
mode=GraderMode.POINTWISE,
template="""
Compare Response A and Response B for the query below.
Which response better addresses the user's question?
Query: {query}
Response A: {response_a}
Response B: {response_b}
Score 1.0 if A is better, 0.0 if B is better, 0.5 if tied.
Respond in JSON: {{"score": <float>, "reason": "<explanation>"}}
""",
)
# Build pairwise dataset with position swap
dataset = []
for sample in test_samples:
# Original order
dataset.append({
"query": sample["query"],
"response_a": baseline_outputs[sample["id"]],
"response_b": candidate_outputs[sample["id"]],
"metadata": {"model_a": "baseline", "model_b": "candidate"},
})
# Swapped order — critical for debiasing
阅读完整来源 ↗