uig
Build a User Input Grid (UIG) for an AI product or feature, evaluate existing eval datasets against it, and propose new inputs to fill coverage gaps. Use this skill whenever someone asks to design an eval framework, audit a test set, build a "user input grid", "synthetic query matrix" or "SQM", improve dataset coverage, or assess whether their offline evals reflect what real users will actually send. Also trigger when they share a product domain + sample dataset and want to know what the dataset is missing, or when they're setting up evals from scratch for a new AI feature. Output is a markdown file (UIG framework + dataset evaluation + recommendations) saved to the current project directory.
Calibre-Labs · 已有静态检查记录
安装与使用
运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。
npx skills add Calibre-Labs/reforge-ai-evals --skill uig技能说明
name: uig description: > Build a User Input Grid (UIG) for an AI product or feature, evaluate existing eval datasets against it, and propose new inputs to fill coverage gaps. Use this skill whenever someone asks to design an eval framework, audit a test set, build a "user input grid", "synthetic query matrix" or "SQM", improve dataset coverage, or assess whether their offline evals reflect what real users will actually send. Also trigger when they share a product domain
- sample dataset and want to know what the dataset is missing, or when they're setting up evals from scratch for a new AI feature. Output is a markdown file (UIG framework + dataset evaluation + recommendations) saved to the current project directory.
User Input Grid Skill
A User Input Grid (UIG) — also called a Synthetic Query Matrix (SQM) — is a structured framework for ensuring eval coverage of an AI product reflects the real diversity of users, content domains, input types, and input styles the system will face in production. It comes from the Reforge AI Evals curriculum and a few teams at Amplitude, Linear, and Notion have published versions of it.
The core idea: pick 3–5 dimensions that drive meaningful diversity in your product, define 4–7 values per dimension, and treat the cartesian product (pruned for plausibility) as your eval sourcing matrix. A 4-dimension grid with average ~5 values per dimension gives ~625 theoretical combinations — practical eval sets land at 30–80 high-coverage inputs after pruning.
The reason this matters: most AI eval datasets are built from one of three weak sources — engineer intuition (overfits to features the engineer knows about), real-user logs (overfits to common easy cases users actually try), or synthetic generation without a framework (random uneven coverage). A UIG forces you to be deliberate about what you're testing, makes gaps visible, and gives you a shared vocabulary with the product team for talking about coverage.
When to use this skill
This skill triggers in three modes — be explicit about which one you're in before starting:
-
Greenfield UIG — designing a UIG for a new AI product/feature with no existing dataset. Workflow ends at "framework + example inputs to seed the dataset."
-
Audit existing dataset(s) — there are one or more eval datasets and you want to know what they're missing. Workflow includes evaluating each dataset against the UIG with STRONG/PARTIAL/WEAK coverage ratings, surfacing gaps, and recommending additions.
-
Both — design the UIG informed by patterns in the existing dataset, then audit the dataset against the UIG. This is the most common case when a team has been building evals ad-hoc and wants to put structure around them.
If the mode isn't specified, ask in one short question, then proceed.
Workflow
Step 1 — Understand the product
Before designing dimensions, you need to know: what does this AI product do, who uses it, what data does it have access to, what tools does it call, and what does success look like? Don't skip this — a UIG built without product understanding produces generic dimensions that don't catch real failure modes.
If the user has shared product docs, sample inputs, system prompts, or tool definitions, read them first. If you don't have enough context, ask 2–3 specific questions — not a generic interview. Examples: "What actions can the AI take? Are there topics it explicitly can't handle? What are the most common user goals?"
Step 2 — Read the dataset(s) if any exist
If there are existing datasets, read them before designing the UIG. Real input data is the best signal you have. Look for:
- Persona signals — are there user-type labels in metadata? Do inputs imply different user contexts (new user, power user, admin, customer vs. employee)?
- Domain coverage — what topic areas do inputs touch? What's missing from the inputs that the product clearly supports?
- Input type distribution — how many action instructions vs. information requests vs. multi-step tasks vs. ambiguous inputs?
- Style patterns — are inputs well-specified, vague, count-only, edge cases?
- Scoring criteria quality — are pass/fail criteria deterministic, or vague LLM-judge prompts? Watch for OR conditions ("X or asks for clarification") that make the test always pass.
This "dataset-first reading" surfaces dimensions you'd never invent from first principles.
Step 3 — Design the dimensions
Pick 3–5 dimensions. Fewer than 3 and the grid doesn't drive enough diversity; more than 5 and the cartesian product explodes and the dimensions start overlapping conceptually.
Two dimensions are nearly universal for any AI product. Use these as defaults, adapted to context:
Input Type — what the user is asking the AI to do
| Value | Definition | Example |
|---|---|---|
| Action instruction | Direct the AI to perform an operation | "Update the Nike opportunity to Closed Won" |
| Information request | Ask the AI to retrieve or explain something | "What's the status of case #12345?" |
| Aggregation | Compute or summarize across a set | "How many open P1 cases do I have this week?" |
| Multi-step | Chain of operations in one input | "Find at-risk accounts and draft outreach emails" |
| Clarification / follow-up | Continue or refine a prior turn | "Actually, make the tone more formal" |
| Ambiguous / under-specified | Intent unclear, AI must resolve | "Handle this" with no supporting context |
| Edge / out-of-scope | Outside the AI's topics or capabilities | A request the system is not designed to fulfill |
For products with workflow-triggered inputs (no user text — an automated record or event fires the AI): add Automated trigger as a value and ensure your eval dataset includes both human-initiated and system-initiated inputs.
For AI products with traditional search/query interfaces, "Action instruction" may be rare a
阅读完整来源 ↗