Auto-Arena: Benchmark AI Models Without Test Data

Where this fits

You need to pick the best model for a niche task—say, extracting structured data from messy invoices—but you have no eval set, no labeled examples, and no time to hand-craft prompts. Instead of manually running side-by-side tests, you delegate the entire benchmarking loop to an agent. auto-arena turns a one-line task description into a full arena: it generates diverse test queries, collects responses from every endpoint, builds evaluation rubrics on the fly, runs pairwise comparisons with a judge model, and hands you a win-rate ranking with charts.

Why agents benefit

• Zero-dependency evaluation: Agents can benchmark models on arbitrary tasks without requiring pre-existing test data or golden answers.
• Rubric generation from context: The skill auto-creates task-specific criteria based on the task description and sample queries, so scoring reflects what actually matters.
• Position-bias handling: Pairwise comparisons are run in both orders and aggregated, reducing judge bias and producing more reliable rankings.
• Checkpoint and incremental support: Long evals can resume from saved state, and you can add new endpoints mid-run without restarting—useful for comparing newly released models.
• Judge hot-swap: If the default judge is biased or unavailable, you can swap in a different model without regenerating prior results.

Practical scenarios

Choosing a coding assistant for a specific codebase

Describe your stack and coding conventions. auto-arena generates questions about refactoring, bug fixing, and API usage, then ranks Claude, GPT, and local models on answer quality and style fit.

Comparing agent frameworks on a multi-step task

Need to know which agent orchestration layer handles tool calls and memory best? Define the task (e.g., "plan a trip with constraints"), and auto-arena evaluates each framework's end-to-end output, not just single responses.

Validating a fine-tuned model against baselines

Before shipping a custom fine-tune, run an arena against the base model and a stronger commercial model. The generated queries cover edge cases you wouldn't think of manually, and the win matrix shows where your model wins and loses.

Add it to your agent workflow

Install the skill into your agent environment:

npx skills add agentscope-ai/OpenJudge --skill auto-arena

Then ask your agent to run a comparison. For example:

Compare gpt-4o-mini, claude-3-haiku, and llama-3.1-8b on writing concise, empathetic customer support replies.

The skill will generate test queries, collect responses, create a rubric, run pairwise judging, and output a Markdown report with a win-rate chart. You can also resume a partial run or add a new model mid-evaluation.

For more details, see the auto-arena skill page.

Compare before adopting

• Quality signals: Check how the skill handles ambiguous tasks—does it generate queries that actually stress the models? Look at the rubric quality and judge consistency.
• Maintenance freshness: The repo (OpenJudge) has 809 stars and active updates; verify recent commits and issue responses.
• Alternatives: You could script your own eval harness, but you'd lose auto-rubric generation and position-bias handling. Consider tools like lm-eval-harness if you have fixed datasets.
• Workflow fit: This skill shines for exploratory comparisons. For production regression testing, you'll still want a curated eval set.

Why it is worth tracking

auto-arena addresses a common pain point—evaluating models on novel tasks without test data—and does it with a practical pipeline that agents can invoke autonomously. The project's star growth and active development signal real community interest. Keep it on your radar if you regularly benchmark models, and evaluate it on your own tasks to see if the auto-generated rubrics match your quality bar. If they do, it becomes a reusable tool for every model selection decision.