vally-eval: Bring rigor to agent skill testing

Where this fits

You have just added a new tool call to your agent skill and updated eval.yaml, but you are not sure whether the agent still completes the target scenario. Instead of manually crafting prompts and eyeballing outputs, you ask the agent to author or update a Vally eval suite, validate it, and run it locally. vally-eval embeds that workflow directly into the agent, so evaluation becomes part of the skill development loop.

Why agents benefit

• The agent can generate spec-compliant eval suites from a natural-language description of the skill scenario, reducing YAML schema mistakes.
• It validates eval.yaml files before execution, catching invalid formatters, missing stimuli, and broken grader configuration.
• It maps and migrates existing tests into Vally suites, turning scattered prompt checks into reproducible eval cases.
• It invokes vally to run the suite and interpret scores, helping agents iterate on grading logic instead of guessing.
• It can add eval execution to CI, making agent-skill regressions visible in pull requests.

Practical scenarios

Authoring a new eval suite from a skill description

A developer asks, "create an eval for the parse-args skill." The agent scaffolds the suite under the correct plugin and skill paths, fills stimuli from known failure cases, and sets up graders.

Debugging a failing grading prompt

The suite runs and an unexpected test fails. The agent inspects the raw agent output, identifies that the rubric is too strict, and updates the grader in eval.yaml before rerunning.

Migrating an existing test harness to Vally

A team has CSV prompts and manual regression notes. vally-eval maps those into eval.yaml stimuli and expected outcomes, so the skill gets the same coverage with less ad-hoc tooling.

Add it to your agent workflow

Install the skill with:

npx skills add microsoft/GitHub-Copilot-for-Azure --skill vally-eval

Then, in a project with a .vally.yaml, prompt the agent:

Add a stimulus for the missing flag case to eval.yaml, validate the suite, and run it with vally.

The skill locates the eval suite, edits the schema, runs validation, and summarizes the pass/fail results.

Compare before adopting

Use vally-eval if you already use Vally or need Azure skills plugin integration. Compare it with plain vally CLI workflows, general-purpose eval tools like promptfoo, and hand-rolled test scripts. Key signals: do you need a maintainer-backed skill? Microsoft's repository has 248 stars and active support. Is the plugin-specific file layout relevant to your repo? And does giving an agent write access to eval files fit your trust boundaries?

Why it is worth tracking

vally-eval is part of a Microsoft-maintained skill collection, which is a strong freshness and quality signal. The trigger vocabulary is narrow but effective for eval-heavy tasks, and the skill is useful the moment you start building integration tests for agent skills. Track it when you need a repeatable way to author, run, and validate skill-level evals. For a catalog view and comparisons, visit https://www.openagentskill.com/skills/microsoft-vally-eval.