跳至正文
OpenAgentSkill
浏览已开放 · 目录记录日期 2026-11-21 · 自动更新和账户服务暂时关闭。
法律与合规

ref-hallucination-arena

Benchmark LLM reference recommendation capabilities by verifying every cited paper against Crossref, PubMed, arXiv, and DBLP. Measures hallucination rate, per-field accuracy (title/author/year/DOI), discipline breakdown, and year constraint compliance. Supports tool-augmented (ReAct + web search) mode. Use when the user asks to evaluate, benchmark, or compare models on academic reference hallucination, literature recommendation quality, or citation accuracy.

agentscope-ai · 已有 AI 审核记录

安装与使用

运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。

npx skills add agentscope-ai/OpenJudge --skill ref-hallucination-arena

技能说明


name: ref-hallucination-arena description: > Benchmark LLM reference recommendation capabilities by verifying every cited paper against Crossref, PubMed, arXiv, and DBLP. Measures hallucination rate, per-field accuracy (title/author/year/DOI), discipline breakdown, and year constraint compliance. Supports tool-augmented (ReAct + web search) mode. Use when the user asks to evaluate, benchmark, or compare models on academic reference hallucination, literature recommendation quality, or citation accuracy.

Reference Hallucination Arena Skill

Evaluate how accurately LLMs recommend real academic references using the OpenJudge RefArenaPipeline:

  1. Load queries — from JSON/JSONL dataset
  2. Collect responses — BibTeX-formatted references from target models
  3. Extract references — parse BibTeX entries from model output
  4. Verify references — cross-check against Crossref / PubMed / arXiv / DBLP
  5. Score & rank — compute verification rate, per-field accuracy, discipline breakdown
  6. Generate report — Markdown report + visualization charts

Prerequisites

# Install OpenJudge
pip install py-openjudge

# Extra dependency for ref_hallucination_arena (chart generation)
pip install matplotlib

Gather from user before running

InfoRequired?Notes
Config YAML pathYesDefines endpoints, dataset, verification settings
Dataset pathYesJSON/JSONL file with queries (can be set in config)
API keysYesEnv vars: OPENAI_API_KEY, DASHSCOPE_API_KEY, etc.
CrossRef emailNoImproves API rate limits for verification
PubMed API keyNoImproves PubMed rate limits
Output directoryNoDefault: ./evaluation_results/ref_hallucination_arena
Report languageNo"en" (default) or "zh"
Tavily API keyNoRequired only if using tool-augmented mode

Quick start

CLI

# Run evaluation with config file
python -m cookbooks.ref_hallucination_arena --config config.yaml --save

# Resume from checkpoint (default behavior)
python -m cookbooks.ref_hallucination_arena --config config.yaml --save

# Start fresh, ignore checkpoint
python -m cookbooks.ref_hallucination_arena --config config.yaml --fresh --save

# Override output directory
python -m cookbooks.ref_hallucination_arena --config config.yaml \
  --output_dir ./my_results --save

Python API

import asyncio
from cookbooks.ref_hallucination_arena.pipeline import RefArenaPipeline

async def main():
    pipeline = RefArenaPipeline.from_config("config.yaml")
    result = await pipeline.evaluate()

    for rank, (model, score) in enumerate(result.rankings, 1):
        print(f"{rank}. {model}: {score:.1%}")

asyncio.run(main())

CLI options

FlagDefaultDescription
--config—Path to YAML configuration file (required)
--output_dirconfig valueOverride output directory
--saveFalseSave results to file
--freshFalseStart fresh, ignore checkpoint

Minimal config file

task:
  description: "Evaluate LLM reference recommendation capabilities"

dataset:
  path: "./data/queries.json"

target_endpoints:
  model_a:
    base_url: "https://api.openai.com/v1"
    api_key: "${OPENAI_API_KEY}"
    model: "gpt-4"
    system_prompt: "You are an academic literature recommendation expert. Recommend {num_refs} real papers in BibTeX format. Only recommend papers you are confident actually exist."

  model_b:
    base_url: "https://dashscope.aliyuncs.com/compatible-mode/v1"
    api_key: "${DASHSCOPE_API_KEY}"
    model: "qwen3-max"
    system_prompt: "You are an academic literature recommendation expert. Recommend {num_refs} real papers in BibTeX format. Only recommend papers you are confident actually exist."

Full config reference

task

FieldRequiredDescription
descriptionYesEvaluation task description
scenarioNoUsage scenario

dataset

FieldDefaultDescription
path—Path to JSON/JSONL dataset file (required)
shufflefalseShuffle queries before evaluation
max_queriesnullMax queries to use (null = all)

target_endpoints.<name>

FieldDefaultDescription
base_url—API base URL (required)
api_key—API key, supports ${ENV_VAR} (required)
model—Model name (required)
system_promptbuilt-inSystem prompt; use {num_refs} placeholder
max_concurrency5Max concurrent requests for this endpoint
extra_params—Extra API request params (e.g. temperature)
tool_config.enabledfalseEnable ReAct agent with Tavily web search
tool_config.tavily_api_keyenv varTavily API key
tool_config.max_iterations10Max ReAct iterations (1–30)
tool_config.search_depth"advanced""basic" or "advanced"

verification

FieldDefaultDescription
crossref_mailto—Email for Crossref polite pool
pubmed_api_key—PubMed API key
max_workers10Concurrent verification threads (1–50)
timeout30Per-request timeout in seconds
verified_threshold0.7Min composite score to count as VERIFIED

evaluation

FieldDefaultDescription
timeout120Model API request timeout in seconds
retry_times3Number of retry attempts

output

FieldDefaultDescription
output_dir./evaluation_results/ref_hallucination_arenaOutput directory
save_queriestrueSave loaded queries
save_responsestrueSave model responses
save_detailstrueSave verification details

阅读完整来源 ↗