create-custom-grader
Use when converting an existing benchmark, rubric, verifier, task YAML/JSON, or domain check into SkillEvaluator BYOG/BYOT custom evaluation.
SkillEvaluator Maintainers <maintainers@example.com> · 已有静态检查记录
安装与使用
运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。
npx skills add NVIDIA/SkillEvaluator --skill create-custom-grader技能说明
name: create-custom-grader description: Use when converting an existing benchmark, rubric, verifier, task YAML/JSON, or domain check into SkillEvaluator BYOG/BYOT custom evaluation. metadata: author: SkillEvaluator Maintainers maintainers@example.com
Create Custom Grader
Convert team-owned benchmark definitions into runnable SkillEvaluator custom graders and, when needed, native Harbor tasks.
Purpose
Help an agent author valid SkillEvaluator BYOG/BYOT files from a user's benchmark instead of leaving the user with empty grader templates.
When To Use
Use this skill when the user wants to:
- bring an existing benchmark into SkillEvaluator
- turn a rubric into
evals/grader.pyorevals/grader.sh - add custom metrics beside the default evaluator metrics
- convert task files such as
task.yaml,task.json, pytest checks, or shell verifiers into BYOG or BYOT - prove a team can run its own benchmark through SkillEvaluator
Do not use this skill for ordinary evals/evals.json authoring when no custom
grading logic is needed. Use the normal dataset authoring workflow for that.
Instructions
- Read the target skill, existing
evals/, benchmark prompts, fixtures, and any verifier code. - Choose
default_plus_customwhen custom metrics should complement default evaluator scoring. - Choose
custom_onlyonly when the user wants the custom grader to own pass/fail semantics. - Write or update
evals/grader.pyorevals/grader.sh, then validate the Harbor contract.
Examples
skillevaluator init-custom-grader <skill-dir> --language python --mode default_plus_custom
skillevaluator tier3 validate <skill-dir>
Prerequisites
- The target skill directory should contain
SKILL.md. - The SkillEvaluator CLI should be available as
skillevaluator. - Full E2E evaluation may need agent credentials, sandbox access, GPU access, or service credentials depending on the benchmark.
Core Choice
Choose one path before writing files:
| User need | Evaluator shape |
|---|---|
Existing evals.json task plus extra domain checks | Top-level BYOG: evals/grader.py or evals/grader.sh |
| Existing benchmark prompt/rubric that can run in the generated workspace | Top-level BYOG plus evals/evals.json and evals/files/ |
| Benchmark owns task layout, setup, service lifecycle, or verifier harness | Native BYOT/BYOG: evals/harbor/<case>/... |
| User wants only custom reward/pass criteria | grading.mode: custom_only |
| User wants default evaluator dimensions plus custom metrics | grading.mode: default_plus_custom |
Default to default_plus_custom unless the user explicitly wants the custom
grader to replace the default evaluator metrics.
Workflow
-
Resolve the target skill and benchmark source. Read the target
SKILL.md, existingevals/, benchmark prompts, fixtures, rubric, reference solution, tags, and any expected trigger/non-trigger metadata. -
Map benchmark fields into evaluator inputs. Use benchmark prompts or prompt variants as
questionentries. Use the target skill asexpected_skill. Put each case's required starter files underevals/files/<case-id>/, and declarefiles: ["evals/files/<case-id>"]on every corresponding eval entry. Do not omitfilesin a multi-case dataset, because omission intentionally stages the entire shared directory for legacy compatibility. Preserve benchmark-specific rubric text in the entry only when the grader needs to read it. -
Scaffold the evaluator contract. For generated tasks:
skillevaluator init-custom-grader <skill-dir> --language python --mode default_plus_customFor shell checks:
skillevaluator init-custom-grader <skill-dir> --language shell --mode default_plus_customFor native Harbor tasks:
skillevaluator init-harbor-task <skill-dir> --case-id <case-id> --with-config -
Replace scaffold placeholders. The custom grader is real executable logic, not metadata. It must read available evidence, compute numeric scores, and write the evaluator reward contract.
-
Validate before running.
skillevaluator validate <skill-dir> --harbor-contractFix missing files, invalid Python, missing reward output, and native Harbor ID mismatches before evaluation.
-
Run the deepest practical proof. Prefer a real with-skill/baseline run. If services, credentials, GPU, or cost block full E2E, state exactly what was validated and what was not.
Grader Contract
Python and shell graders run inside the Harbor verifier context. They may read:
/logs/agent/trajectory.jsonfor agent actions and final answer evidence/tests/entry.jsonfor the eval case metadata/workspace/input/for the entry's declared committed fixtures fromevals/files//solution/or other task outputs only when the task environment produces them
They must write:
/logs/verifier/reward.json/logs/verifier/reward.txtwith a numeric score from0.0to1.0
Use this reward shape:
{
"overall": 0.92,
"custom_metrics": {
"domain_repair": 1.0,
"domain_verification": 0.8
},
"details": {
"domain_repair": {
"score": 1.0,
"reason": "The solution repaired the required files."
}
}
}
In default_plus_custom, default evaluator scoring keeps its overall
authoritative and adds the grader's custom_metrics into reports. In
custom_only, the grader's overall is the pass/fail reward.
Never emit custom metric names that collide with reserved evaluator fields:
security, skill_execution, skill_efficiency, accuracy,
goal_accuracy, behavior_check, overall, details, metrics,
metric_set, or entry_id.
Translation Rules
- Convert each rubric item into a deterministic check when possible.
- If a rubric item requires judgment, encode observable proxies and explain the
limits in
details. - Keep