eval-dataset
Use when the user asks to turn a failure into an eval, create/review/accept/reject dataset cases, or convert Galileo traces, metric gaps, or production examples into cases.
Galileo-Agent-Labs · Static checks on file
Installation
Read the source and dependencies before running a command. Copying does not install or execute anything.
npx skills add Galileo-Agent-Labs/eval-engineer --skill eval-datasetAbout this skill
name: eval-dataset description: Use when the user asks to turn a failure into an eval, create/review/accept/reject dataset cases, or convert Galileo traces, metric gaps, or production examples into cases.
Eval Dataset
Use this skill to build durable eval cases from evidence. Its job is dataset quality control, not diagnosis or app fixing.
Required Reference
Use skills/eval-engineer/references/eval-datasets.md for the canonical case
schema precedence, optional review metadata, promotion rules, bootstrap
guidance for different use cases, and Galileo SDK dataset usage.
Required Inputs
Start from at least one evidence source:
.galileo/current/debug-packet.json.galileo/current/diagnosis.md- Galileo trace, session, experiment, or log-stream IDs
- production symptom with enough context to define expected behavior
If there is no concrete failure, regression, policy requirement, or metric gap, ask for evidence before writing a case.
Do
- Write new unreviewed cases to
.galileo/eval-dataset/candidates.jsonlunless the user asks to accept or reject a case. - When accepting or rejecting cases, update
.galileo/eval-dataset/changelog.md. - Bootstrap datasets by use case: RAG, tool-calling agent, multi-turn, workflow, safety/compliance, and tokenomics.
- Choose failure triggers that should be caught by named Galileo metrics or explicit local gates.
- Follow the user-provided schema when the user gives one.
- Follow the existing Galileo dataset schema when appending to a fixed dataset.
- Do not force Eval Engineer fields into an upload schema. Put optional review
metadata in
.galileo/eval-dataset/candidates.jsonl,metadata, or a sidecar file when needed. - Use retrieved-source gates when source authority matters and the chosen schema can carry them. Final citations alone are not enough for prompt-injection, privacy, stale-source, or source-authority cases.
- When the user wants to upload, reuse, or run cases in Galileo, use the SDK workflow in the reference instead of improvising dataset shapes.
- Do not promote candidates without human review.
Validation Loop
Before finalizing a dataset change:
- Check the case against
skills/eval-engineer/references/eval-datasets.md. - Confirm the case has a concrete failure trigger and metric or local gate that should catch it.
- Confirm required/forbidden citations, tools, permissions, source gates, and abstention rules are explicit when relevant.
- Validate JSONL syntax after writing or editing.
- If accepting or rejecting a case, update
.galileo/eval-dataset/changelog.md.
Output
State whether the result is candidate, accepted, rejected, or blocked. Include the target JSONL path, case IDs changed, evidence source, and remaining metric gaps.