Meta-Eval: The Triage Desk for LLM Evaluation

Where this fits

You've built an agent that answers customer questions, but you have no idea if it's actually good. You have logs, a few prompts, maybe a RAG pipeline. You know you need evaluation, but every tutorial assumes you already have a golden set and metrics. That's where meta-eval steps in. It acts as a triage desk: it asks what data you have and how it's labeled, then points you to the right sub-skill without forcing you to design an entire eval system from scratch.

Why agents benefit

• Targeted diagnosis: meta-eval asks two critical questions—data_form (traces, prompts, RAG, or nothing) and label_status—before recommending anything, so you get a route that actually matches your starting point.
• Provisional routing: A hard gate ensures you get a recommendation as soon as those two signals are known, even if you haven't thought about stakes or user priorities yet. You can refine later.
• Self-contained sub-skills: Each downstream workflow carries its own data shapes, statistics, and principles. You don't need to re-learn context or jump between docs.
• Handles the "nothing at all" case: If you have zero traces or labels, meta-eval guides you to a workflow that helps you bootstrap a minimal evaluation setup.
• Keeps the agent focused: Instead of trying to build a full evaluation system, the agent stays in triage mode, which reduces complexity and speeds up iteration.

Practical scenarios

You have traces but no labels

You've been logging agent interactions but haven't annotated any. meta-eval identifies data_form=traces and label_status=unlabeled, then routes you to a sub-skill that helps you create a golden set from those traces—without requiring manual labeling of every example.

You're comparing two prompts

You have a prompt template and want to test two variants. meta-eval sees you're not building a full eval system, just comparing prompts. It recommends a lightweight workflow that sets up side-by-side runs and scores them, skipping the heavy infrastructure.

You have a RAG pipeline and need retrieval accuracy

Your RAG system returns documents, but you're unsure if the retriever is pulling the right ones. meta-eval asks about your data format and label status, then points to a RAG-specific sub-skill that measures retrieval quality and answer faithfulness, so you can isolate the bottleneck.

Add it to your agent workflow

Install the skill with:

npx skills add agentscope-ai/OpenJudge --skill meta-eval

Once installed, invoke it naturally. For example, say: "I need to evaluate my agent but don't know where to start." The skill will ask about your data format and label status, then recommend a sub-skill like eval-from-traces or prompt-comparison. You can then follow that sub-skill's instructions to build your evaluation.

Compare before adopting

Before committing, compare meta-eval against alternatives like building your own eval harness or using a commercial platform. Check quality signals: how many sub-skills does it cover? Is the README current and detailed? Look at community momentum—stars, issues, and recent commits. Also consider workflow fit: does it integrate with your existing agent stack, or would you need to adapt your data pipeline? Meta-eval is an entry router, so its value depends on the quality of the sub-skills it points to.

Why it is worth tracking

With 809 stars and active maintenance from agentscope-ai, meta-eval shows community trust and ongoing evolution. It's the front door to a larger eval skill collection, so tracking it signals whether that ecosystem is growing and improving. Use it on a real project, then evaluate: did it reduce the time to a working eval? Did it ask the right questions? If yes, it's worth keeping in your agent's toolkit. For a broader look at agent skills, see the OpenAgentSkill directory at https://www.openagentskill.com/skills/agentscope-ai-meta-eval.