--- name: rl-reward description: > Build RL reward signals using the OpenJudge framework. Covers choosing between pointwise and pairwise reward strategies based on RL algorithm, task type, and cost; aggregating multi-dimensional pointwise scores into a scalar reward; pairwise tournament reward for GRPO on subjective tasks (net win rate across group rollouts); generating preference pairs for DPO/RLAIF; and normalizing scores for training stability. Use when building reward models, scoring rollouts for GRPO/REINFORCE, generating preference data for DPO, or doing Best-of-N selection. ---

# RL Reward Construction with OpenJudge

Build reward signals for reinforcement learning from human feedback (RLHF) and reinforcement learning from AI feedback (RLAIF) using the `openjudge` library.

## When to Use This Skill

- Building scalar rewards for GRPO / REINFORCE rollout scoring - Generating (chosen, rejected) preference pairs for DPO / IPO - Best-of-N candidate selection - Multi-dimensional reward shaping (correctness + safety + format) - Replacing or bootstrapping a reward model with LLM-as-judge

## Step 1 — Choose Your Reward Strategy

Use this decision tree **before** writing any code:

``` RL Algorithm + Task type? │ ├── GRPO / REINFORCE — Verifiable task (math, code, structured output) │ └── → POINTWISE ✅ (FunctionGrader, exact score, zero LLM cost) │ ├── GRPO / REINFORCE — Subjective task (instruction following, dialogue, summarization) │ └── → PAIRWISE TOURNAMENT ✅ (compare each rollout vs all others in group, │ reward = net win rate within group) │ ├── DPO / IPO / SLiC — need (chosen, rejected) pairs │ └── → PAIRWISE ✅ (two-way comparison, return winner/loser) │ └── Best-of-N / reranking — rank N candidates └── → LISTWISE ✅ (single call ranks all N at once) ```

``` Cost constraint? ├── Low budget │ └── FunctionGrader (free) → pointwise; or pairwise with small judge model │ ├── Medium budget │ └── Pointwise: 2–3 LLM graders + WeightedSumAggregator │ └── Pairwise tournament: 1 LLM judge, N*(N-1)/2 comparisons per group │ └── High quality / no cost limit └── Pointwise voting (3–5 calls) or pairwise with strong judge + debiasing ```

## Sub-documents — Read When Relevant

| Topic | File | Read when… | |-------|------|------------| | Pointwise multi-dim reward | `pointwise.md` | GRPO on verifiable tasks; multi-dimension scoring | | Pairwise reward | `pairwise.md` | GRPO on subjective tasks (tournament); DPO/RLAIF preference pairs |

Read the relevant sub-document **before** writing any code.

## Install

```bash pip install py-openjudge ```

## Strategy Comparison

| Strategy | Output | Reward signal | Typical use | Cost | |----------|--------|---------------|-------------|------| | **Pointwise** | scalar per response | direct reward `r(x, y)` | GRPO on verifiable tasks, filtering | Low–Medium | | **Pairwise Tournament** | net win rate per response | relative reward within group | GRPO on subjective tasks | Medium (N²/2 calls) | | **Pairwise** | winner/loser pair | implicit preference `y+ > y-` | DPO, IPO, RLAIF preference data | Medium | | **Listwise** | rank over N responses | ordinal reward / reranking | Best-of-N, reranking | Medium–High |

## Score Normalization

All graders return scores on different scales. **Always normalize** before feeding into RL:

```python def normalize(score: float, min_score: float, max_score: float) -> float: """Map [min_score, max_score] → [0.0, 1.0].""" if max_score == min_score: return 0.0 return (score - min_score) / (max_score - min_score)

# LLM graders (common/*) return 1–5 → normalize to 0–1 reward = normalize(result.score, min_score=1, max_score=5)

# FunctionGrader / text graders already return 0–1 → no normalization needed ```

## Evaluation Strategies

Evaluation strategies control **how many times** a grader is called and **how results are aggregated**. They are independent of the grader itself.

### Choose Your Strategy

``` Grader type? │ ├── Deterministic (FunctionGrader, StringMatch, CodeExecution, etc.) │ └── → Direct (zero variance, no need for aggregation) │ ├── LLM grader — Pointwise scoring │ │ │ ├── Budget limited / speed critical │ │ └── → Direct (accept variance, 1× cost) │ │ │ ├── Discrete scores (1–5 integer, pass/fail, binary) │ │ └── → Voting (majority vote, robust to outliers, N× cost) │ │ │ └── Continuous / fine-grained scores (need precise ranking) │ └── → Average (mean, preserves signal, N× cost) │ └── LLM grader — Pairwise GRPO tournament └── → GRPOTournament (all-pairs comparison, net win rate) ```

| Strategy | Aggregation | Best for | Cost | |----------|-------------|----------|------| | `DirectEvaluationStrategy` | None | Deterministic graders; low budget | 1× | | `VotingEvaluationStrategy` | Majority vote | Discrete / integer LLM scores | N× | | `AverageEvaluationStrategy` | Mean | Continuous LLM scores | N× | | `GRPOTournamentEvaluationStrategy` | Net win rate | Pairwise GRPO on subjective tasks | N²/2× |

All strategies are imported from `openjudge.evaluation_strategy`.

### Pointwise — Noise Reduction with Voting / Average

For high-variance LLM judges, wrap any grader with `VotingEvaluationStrategy` to run N calls and take the majority vote:

```python from openjudge.evaluation_strategy import VotingEvaluationStrategy

grader = CorrectnessGrader( model=model, strategy=VotingEvaluationStrategy(num_votes=3, tie_breaker="closest_to_mean"), ) # Now each call internally runs 3 LLM evaluations and returns the most common score ```

Use odd `num_votes` (3, 5) to avoid ties.

### Pairwise — GRPO Tournament

For GRPO on subjective tasks, use `GRPOTournamentEvaluationStrategy` to run all-pairs comparison and compute net win rate per rollout:

```python from openjudge.evaluation_strategy import GRPOTournamentEvaluationStrategy

strategy = GRPOTournamentEvaluationStrategy(debiased=False) results = await strategy.execute( pairwise_grader.aevaluate, query="Write a haiku about the ocean.", responses=["rollout_1", "rollout_2", "rollout_3", "rollout_4"], ) rewards = [r.score for r in results] # net win rates in [-1.0, 1.0] ```

Set `debiased=True` to run each pair in both orders and only count consistent results (doubles LLM calls but mitigates position bias).

--- name: rl-reward description: > Build RL reward signals using the OpenJudge framework. Covers choosing between pointwise and pairwise reward strategies based on RL algorithm, task type, and cost; aggregating multi-dimensional pointwise scores into a scalar reward; pairwise tournament reward for GRPO on subjective tasks (net win rate across group rollouts); generating preference pairs for DPO/RLAIF; and normalizing scores for training stability. Use when building reward models, scoring rollouts for GRPO/REINFORCE, generating preference data for DPO, or doing Best-of-N selection. ---

# RL Reward Construction with OpenJudge

Build reward signals for reinforcement learning from human feedback (RLHF) and reinforcement learning from AI feedback (RLAIF) using the `openjudge` library.

## When to Use This Skill

- Building scalar rewards for GRPO / REINFORCE rollout scoring - Generating (chosen, rejected) preference pairs for DPO / IPO - Best-of-N candidate selection - Multi-dimensional reward shaping (correctness + safety + format) - Replacing or bootstrapping a reward model with LLM-as-judge

## Step 1 — Choose Your Reward Strategy

Use this decision tree **before** writing any code:

``` RL Algorithm + Task type? │ ├── GRPO / REINFORCE — Verifiable task (math, code, structured output) │ └── → POINTWISE ✅ (FunctionGrader, exact score, zero LLM cost) │ ├── GRPO / REINFORCE — Subjective task (instruction following, dialogue, summarization) │ └── → PAIRWISE TOURNAMENT ✅ (compare each rollout vs all others in group, │ reward = net win rate within group) │ ├── DPO / IPO / SLiC — need (chosen, rejected) pairs │ └── → PAIRWISE ✅ (two-way comparison, return winner/loser) │ └── Best-of-N / reranking — rank N candidates └── → LISTWISE ✅ (single call ranks all N at once) ```

``` Cost constraint? ├── Low budget │ └── FunctionGrader (free) → pointwise; or pairwise with small judge model │ ├── Medium budget │ └── Pointwise: 2–3 LLM graders + WeightedSumAggregator │ └── Pairwise tournament: 1 LLM judge, N*(N-1)/2 comparisons per group │ └── High quality / no cost limit └── Pointwise voting (3–5 calls) or pairwise with strong judge + debiasing ```

## Sub-documents — Read When Relevant

| Topic | File | Read when… | |-------|------|------------| | Pointwise multi-dim reward | `pointwise.md` | GRPO on verifiable tasks; multi-dimension scoring | | Pairwise reward | `pairwise.md` | GRPO on subjective tasks (tournament); DPO/RLAIF preference pairs |

Read the relevant sub-document **before** writing any code.

## Install

```bash pip install py-openjudge ```

## Strategy Comparison

| Strategy | Output | Reward signal | Typical use | Cost | |----------|--------|---------------|-------------|------| | **Pointwise** | scalar per response | direct reward `r(x, y)` | GRPO on verifiable tasks, filtering | Low–Medium | | **Pairwise Tournament** | net win rate per response | relative reward within group | GRPO on subjective tasks | Medium (N²/2 calls) | | **Pairwise** | winner/loser pair | implicit preference `y+ > y-` | DPO, IPO, RLAIF preference data | Medium | | **Listwise** | rank over N responses | ordinal reward / reranking | Best-of-N, reranking | Medium–High |

## Score Normalization

All graders return scores on different scales. **Always normalize** before feeding into RL:

```python def normalize(score: float, min_score: float, max_score: float) -> float: """Map [min_score, max_score] → [0.0, 1.0].""" if max_score == min_score: return 0.0 return (score - min_score) / (max_score - min_score)

# LLM graders (common/*) return 1–5 → normalize to 0–1 reward = normalize(result.score, min_score=1, max_score=5)

# FunctionGrader / text graders already return 0–1 → no normalization needed ```

## Evaluation Strategies

Evaluation strategies control **how many times** a grader is called and **how results are aggregated**. They are independent of the grader itself.

### Choose Your Strategy

``` Grader type? │ ├── Deterministic (FunctionGrader, StringMatch, CodeExecution, etc.) │ └── → Direct (zero variance, no need for aggregation) │ ├── LLM grader — Pointwise scoring │ │ │ ├── Budget limited / speed critical │ │ └── → Direct (accept variance, 1× cost) │ │ │ ├── Discrete scores (1–5 integer, pass/fail, binary) │ │ └── → Voting (majority vote, robust to outliers, N× cost) │ │ │ └── Continuous / fine-grained scores (need precise ranking) │ └── → Average (mean, preserves signal, N× cost) │ └── LLM grader — Pairwise GRPO tournament └── → GRPOTournament (all-pairs comparison, net win rate) ```

| Strategy | Aggregation | Best for | Cost | |----------|-------------|----------|------| | `DirectEvaluationStrategy` | None | Deterministic graders; low budget | 1× | | `VotingEvaluationStrategy` | Majority vote | Discrete / integer LLM scores | N× | | `AverageEvaluationStrategy` | Mean | Continuous LLM scores | N× | | `GRPOTournamentEvaluationStrategy` | Net win rate | Pairwise GRPO on subjective tasks | N²/2× |

All strategies are imported from `openjudge.evaluation_strategy`.

### Pointwise — Noise Reduction with Voting / Average

For high-variance LLM judges, wrap any grader with `VotingEvaluationStrategy` to run N calls and take the majority vote:

```python from openjudge.evaluation_strategy import VotingEvaluationStrategy

grader = CorrectnessGrader( model=model, strategy=VotingEvaluationStrategy(num_votes=3, tie_breaker="closest_to_mean"), ) # Now each call internally runs 3 LLM evaluations and returns the most common score ```

Use odd `num_votes` (3, 5) to avoid ties.

### Pairwise — GRPO Tournament

For GRPO on subjective tasks, use `GRPOTournamentEvaluationStrategy` to run all-pairs comparison and compute net win rate per rollout:

```python from openjudge.evaluation_strategy import GRPOTournamentEvaluationStrategy

strategy = GRPOTournamentEvaluationStrategy(debiased=False) results = await strategy.execute( pairwise_grader.aevaluate, query="Write a haiku about the ocean.", responses=["rollout_1", "rollout_2", "rollout_3", "rollout_4"], ) rewards = [r.score for r in results] # net win rates in [-1.0, 1.0] ```

Set `debiased=True` to run each pair in both orders and only count consistent results (doubles LLM calls but mitigates position bias).

--- name: rl-reward description: > Build RL reward signals using the OpenJudge framework. Covers choosing between pointwise and pairwise reward strategies based on RL algorithm, task type, and cost; aggregating multi-dimensional pointwise scores into a scalar reward; pairwise tournament reward for GRPO on subjective tasks (net win rate across group rollouts); generating preference pairs for DPO/RLAIF; and normalizing scores for training stability. Use when building reward models, scoring rollouts for GRPO/REINFORCE, generating preference data for DPO, or doing Best-of-N selection. ---

# RL Reward Construction with OpenJudge

Build reward signals for reinforcement learning from human feedback (RLHF) and reinforcement learning from AI feedback (RLAIF) using the `openjudge` library.

## When to Use This Skill

- Building scalar rewards for GRPO / REINFORCE rollout scoring - Generating (chosen, rejected) preference pairs for DPO / IPO - Best-of-N candidate selection - Multi-dimensional reward shaping (correctness + safety + format) - Replacing or bootstrapping a reward model with LLM-as-judge

## Step 1 — Choose Your Reward Strategy

Use this decision tree **before** writing any code:

``` RL Algorithm + Task type? │ ├── GRPO / REINFORCE — Verifiable task (math, code, structured output) │ └── → POINTWISE ✅ (FunctionGrader, exact score, zero LLM cost) │ ├── GRPO / REINFORCE — Subjective task (instruction following, dialogue, summarization) │ └── → PAIRWISE TOURNAMENT ✅ (compare each rollout vs all others in group, │ reward = net win rate within group) │ ├── DPO / IPO / SLiC — need (chosen, rejected) pairs │ └── → PAIRWISE ✅ (two-way comparison, return winner/loser) │ └── Best-of-N / reranking — rank N candidates └── → LISTWISE ✅ (single call ranks all N at once) ```

``` Cost constraint? ├── Low budget │ └── FunctionGrader (free) → pointwise; or pairwise with small judge model │ ├── Medium budget │ └── Pointwise: 2–3 LLM graders + WeightedSumAggregator │ └── Pairwise tournament: 1 LLM judge, N*(N-1)/2 comparisons per group │ └── High quality / no cost limit └── Pointwise voting (3–5 calls) or pairwise with strong judge + debiasing ```

## Sub-documents — Read When Relevant

| Topic | File | Read when… | |-------|------|------------| | Pointwise multi-dim reward | `pointwise.md` | GRPO on verifiable tasks; multi-dimension scoring | | Pairwise reward | `pairwise.md` | GRPO on subjective tasks (tournament); DPO/RLAIF preference pairs |

Read the relevant sub-document **before** writing any code.

## Install

```bash pip install py-openjudge ```

## Strategy Comparison

| Strategy | Output | Reward signal | Typical use | Cost | |----------|--------|---------------|-------------|------| | **Pointwise** | scalar per response | direct reward `r(x, y)` | GRPO on verifiable tasks, filtering | Low–Medium | | **Pairwise Tournament** | net win rate per response | relative reward within group | GRPO on subjective tasks | Medium (N²/2 calls) | | **Pairwise** | winner/loser pair | implicit preference `y+ > y-` | DPO, IPO, RLAIF preference data | Medium | | **Listwise** | rank over N responses | ordinal reward / reranking | Best-of-N, reranking | Medium–High |

## Score Normalization

All graders return scores on different scales. **Always normalize** before feeding into RL:

```python def normalize(score: float, min_score: float, max_score: float) -> float: """Map [min_score, max_score] → [0.0, 1.0].""" if max_score == min_score: return 0.0 return (score - min_score) / (max_score - min_score)

# LLM graders (common/*) return 1–5 → normalize to 0–1 reward = normalize(result.score, min_score=1, max_score=5)

# FunctionGrader / text graders already return 0–1 → no normalization needed ```

## Evaluation Strategies

Evaluation strategies control **how many times** a grader is called and **how results are aggregated**. They are independent of the grader itself.

### Choose Your Strategy

``` Grader type? │ ├── Deterministic (FunctionGrader, StringMatch, CodeExecution, etc.) │ └── → Direct (zero variance, no need for aggregation) │ ├── LLM grader — Pointwise scoring │ │ │ ├── Budget limited / speed critical │ │ └── → Direct (accept variance, 1× cost) │ │ │ ├── Discrete scores (1–5 integer, pass/fail, binary) │ │ └── → Voting (majority vote, robust to outliers, N× cost) │ │ │ └── Continuous / fine-grained scores (need precise ranking) │ └── → Average (mean, preserves signal, N× cost) │ └── LLM grader — Pairwise GRPO tournament └── → GRPOTournament (all-pairs comparison, net win rate) ```

| Strategy | Aggregation | Best for | Cost | |----------|-------------|----------|------| | `DirectEvaluationStrategy` | None | Deterministic graders; low budget | 1× | | `VotingEvaluationStrategy` | Majority vote | Discrete / integer LLM scores | N× | | `AverageEvaluationStrategy` | Mean | Continuous LLM scores | N× | | `GRPOTournamentEvaluationStrategy` | Net win rate | Pairwise GRPO on subjective tasks | N²/2× |

All strategies are imported from `openjudge.evaluation_strategy`.

### Pointwise — Noise Reduction with Voting / Average

For high-variance LLM judges, wrap any grader with `VotingEvaluationStrategy` to run N calls and take the majority vote:

```python from openjudge.evaluation_strategy import VotingEvaluationStrategy

grader = CorrectnessGrader( model=model, strategy=VotingEvaluationStrategy(num_votes=3, tie_breaker="closest_to_mean"), ) # Now each call internally runs 3 LLM evaluations and returns the most common score ```

Use odd `num_votes` (3, 5) to avoid ties.

### Pairwise — GRPO Tournament

For GRPO on subjective tasks, use `GRPOTournamentEvaluationStrategy` to run all-pairs comparison and compute net win rate per rollout:

```python from openjudge.evaluation_strategy import GRPOTournamentEvaluationStrategy

strategy = GRPOTournamentEvaluationStrategy(debiased=False) results = await strategy.execute( pairwise_grader.aevaluate, query="Write a haiku about the ocean.", responses=["rollout_1", "rollout_2", "rollout_3", "rollout_4"], ) rewards = [r.score for r in results] # net win rates in [-1.0, 1.0] ```

Set `debiased=True` to run each pair in both orders and only count consistent results (doubles LLM calls but mitigates position bias).

--- name: rl-reward description: > Build RL reward signals using the OpenJudge framework. Covers choosing between pointwise and pairwise reward strategies based on RL algorithm, task type, and cost; aggregating multi-dimensional pointwise scores into a scalar reward; pairwise tournament reward for GRPO on subjective tasks (net win rate across group rollouts); generating preference pairs for DPO/RLAIF; and normalizing scores for training stability. Use when building reward models, scoring rollouts for GRPO/REINFORCE, generating preference data for DPO, or doing Best-of-N selection. ---

# RL Reward Construction with OpenJudge

Build reward signals for reinforcement learning from human feedback (RLHF) and reinforcement learning from AI feedback (RLAIF) using the `openjudge` library.

## When to Use This Skill

- Building scalar rewards for GRPO / REINFORCE rollout scoring - Generating (chosen, rejected) preference pairs for DPO / IPO - Best-of-N candidate selection - Multi-dimensional reward shaping (correctness + safety + format) - Replacing or bootstrapping a reward model with LLM-as-judge

## Step 1 — Choose Your Reward Strategy

Use this decision tree **before** writing any code:

``` RL Algorithm + Task type? │ ├── GRPO / REINFORCE — Verifiable task (math, code, structured output) │ └── → POINTWISE ✅ (FunctionGrader, exact score, zero LLM cost) │ ├── GRPO / REINFORCE — Subjective task (instruction following, dialogue, summarization) │ └── → PAIRWISE TOURNAMENT ✅ (compare each rollout vs all others in group, │ reward = net win rate within group) │ ├── DPO / IPO / SLiC — need (chosen, rejected) pairs │ └── → PAIRWISE ✅ (two-way comparison, return winner/loser) │ └── Best-of-N / reranking — rank N candidates └── → LISTWISE ✅ (single call ranks all N at once) ```

``` Cost constraint? ├── Low budget │ └── FunctionGrader (free) → pointwise; or pairwise with small judge model │ ├── Medium budget │ └── Pointwise: 2–3 LLM graders + WeightedSumAggregator │ └── Pairwise tournament: 1 LLM judge, N*(N-1)/2 comparisons per group │ └── High quality / no cost limit └── Pointwise voting (3–5 calls) or pairwise with strong judge + debiasing ```

## Sub-documents — Read When Relevant

| Topic | File | Read when… | |-------|------|------------| | Pointwise multi-dim reward | `pointwise.md` | GRPO on verifiable tasks; multi-dimension scoring | | Pairwise reward | `pairwise.md` | GRPO on subjective tasks (tournament); DPO/RLAIF preference pairs |

Read the relevant sub-document **before** writing any code.

## Install

```bash pip install py-openjudge ```

## Strategy Comparison

| Strategy | Output | Reward signal | Typical use | Cost | |----------|--------|---------------|-------------|------| | **Pointwise** | scalar per response | direct reward `r(x, y)` | GRPO on verifiable tasks, filtering | Low–Medium | | **Pairwise Tournament** | net win rate per response | relative reward within group | GRPO on subjective tasks | Medium (N²/2 calls) | | **Pairwise** | winner/loser pair | implicit preference `y+ > y-` | DPO, IPO, RLAIF preference data | Medium | | **Listwise** | rank over N responses | ordinal reward / reranking | Best-of-N, reranking | Medium–High |

## Score Normalization

All graders return scores on different scales. **Always normalize** before feeding into RL:

```python def normalize(score: float, min_score: float, max_score: float) -> float: """Map [min_score, max_score] → [0.0, 1.0].""" if max_score == min_score: return 0.0 return (score - min_score) / (max_score - min_score)

# LLM graders (common/*) return 1–5 → normalize to 0–1 reward = normalize(result.score, min_score=1, max_score=5)

# FunctionGrader / text graders already return 0–1 → no normalization needed ```

## Evaluation Strategies

Evaluation strategies control **how many times** a grader is called and **how results are aggregated**. They are independent of the grader itself.

### Choose Your Strategy

``` Grader type? │ ├── Deterministic (FunctionGrader, StringMatch, CodeExecution, etc.) │ └── → Direct (zero variance, no need for aggregation) │ ├── LLM grader — Pointwise scoring │ │ │ ├── Budget limited / speed critical │ │ └── → Direct (accept variance, 1× cost) │ │ │ ├── Discrete scores (1–5 integer, pass/fail, binary) │ │ └── → Voting (majority vote, robust to outliers, N× cost) │ │ │ └── Continuous / fine-grained scores (need precise ranking) │ └── → Average (mean, preserves signal, N× cost) │ └── LLM grader — Pairwise GRPO tournament └── → GRPOTournament (all-pairs comparison, net win rate) ```

| Strategy | Aggregation | Best for | Cost | |----------|-------------|----------|------| | `DirectEvaluationStrategy` | None | Deterministic graders; low budget | 1× | | `VotingEvaluationStrategy` | Majority vote | Discrete / integer LLM scores | N× | | `AverageEvaluationStrategy` | Mean | Continuous LLM scores | N× | | `GRPOTournamentEvaluationStrategy` | Net win rate | Pairwise GRPO on subjective tasks | N²/2× |

All strategies are imported from `openjudge.evaluation_strategy`.

### Pointwise — Noise Reduction with Voting / Average

For high-variance LLM judges, wrap any grader with `VotingEvaluationStrategy` to run N calls and take the majority vote:

```python from openjudge.evaluation_strategy import VotingEvaluationStrategy

grader = CorrectnessGrader( model=model, strategy=VotingEvaluationStrategy(num_votes=3, tie_breaker="closest_to_mean"), ) # Now each call internally runs 3 LLM evaluations and returns the most common score ```

Use odd `num_votes` (3, 5) to avoid ties.

### Pairwise — GRPO Tournament

For GRPO on subjective tasks, use `GRPOTournamentEvaluationStrategy` to run all-pairs comparison and compute net win rate per rollout:

```python from openjudge.evaluation_strategy import GRPOTournamentEvaluationStrategy

strategy = GRPOTournamentEvaluationStrategy(debiased=False) results = await strategy.execute( pairwise_grader.aevaluate, query="Write a haiku about the ocean.", responses=["rollout_1", "rollout_2", "rollout_3", "rollout_4"], ) rewards = [r.score for r in results] # net win rates in [-1.0, 1.0] ```

Set `debiased=True` to run each pair in both orders and only count consistent results (doubles LLM calls but mitigates position bias).