跳至正文
OpenAgentSkill
浏览已开放 · 目录记录日期 2026-11-21 · 自动更新和账户服务暂时关闭。
开发与测试

skillgrade-graders

Authors deterministic and LLM rubric graders for skillgrade evaluations. Use when creating scoring scripts, writing evaluation rubrics, or combining multiple graders with weighted scoring. Don't use for setting up eval pipelines, configuring eval.yaml defaults, or general test writing.

mgechev · 已有 AI 审核记录

安装与使用

运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。

npx skills add mgechev/skillgrade --skill skillgrade-graders

技能说明


name: skillgrade-graders description: Authors deterministic and LLM rubric graders for skillgrade evaluations. Use when creating scoring scripts, writing evaluation rubrics, or combining multiple graders with weighted scoring. Don't use for setting up eval pipelines, configuring eval.yaml defaults, or general test writing.

Skillgrade Grader Authoring

Procedures

Step 1: Identify the Grading Strategy

  1. Determine whether the task requires objective verification (deterministic) or qualitative assessment (LLM rubric).
  2. For most tasks, combine both: deterministic graders verify outcomes (weight 0.7), LLM rubrics assess approach quality (weight 0.3).

Step 2: Write a Deterministic Grader

  1. Create a script in the skill's graders/ directory (bash or TypeScript).
  2. The script must output a JSON object to stdout with the following structure:
    {"score": 0.67, "details": "2/3 checks passed", "checks": [{"name": "check-name", "passed": true, "message": "Description"}]}
    
  3. score (0.0–1.0) and details are required. checks is optional but recommended.
  4. Read references/grader-output-schema.md for the full output specification.
  5. Use awk for arithmetic in bash scripts — bc is not available in node:20-slim.
  6. Reference the grader in eval.yaml:
    - type: deterministic
      run: bash graders/check.sh
      weight: 0.7
    

Step 3: Write an LLM Rubric Grader

  1. Draft a rubric with explicit scoring criteria and point allocations.
  2. Structure the rubric into weighted sections that sum to 1.0:
    Workflow Compliance (0-0.5):
    - Did the agent follow the mandatory workflow steps?
    Efficiency (0-0.5):
    - Completed in ≤5 commands without trial-and-error?
    
  3. Reference the rubric in eval.yaml:
    - type: llm_rubric
      rubric: |
        [rubric text or file path]
      weight: 0.3
      provider: gemini               # optional: gemini (default) | anthropic | openai
      model: gemini-3.5-flash        # optional model override (defaults to the latest dynamically resolved flash model)
    
  4. For long rubrics, store in a separate file and reference by path: rubric: rubrics/quality.md.

Step 4: Combine Multiple Graders

  1. Assign weights to each grader based on importance. Weights are normalized automatically.
  2. Final reward is calculated as: Σ (grader_score × weight) / Σ weight.
  3. Example configuration:
    graders:
      - type: deterministic
        run: bash graders/check.sh
        weight: 0.7
      - type: llm_rubric
        rubric: rubrics/quality.md
        weight: 0.3
    

Step 5: Validate Graders

  1. Create a reference solution script that produces the expected output.
  2. Run skillgrade --validate to verify graders score the reference solution correctly.
  3. Test only deterministic graders: skillgrade --grader=deterministic (skips LLM calls, faster iteration).
  4. Test only LLM rubric graders: skillgrade --grader=llm_rubric.
  5. Run a specific eval with a specific grader type: skillgrade --eval=my-eval --grader=deterministic.
  6. If a grader returns unexpected scores, inspect the script output and adjust scoring logic.

Error Handling

  • If a deterministic grader outputs non-JSON, ensure all echo/console.log statements except the final JSON result are redirected to stderr.
  • If an LLM rubric grader returns 0.00 with a missing API key message, set the appropriate key for your provider: GEMINI_API_KEY (provider: gemini), ANTHROPIC_API_KEY (provider: anthropic), or OPENAI_API_KEY (provider: openai).
  • To use a custom/self-hosted LLM endpoint, set ANTHROPIC_BASE_URL (for provider: anthropic) or OPENAI_BASE_URL (for provider: openai) — e.g. for Ollama or vLLM.
  • If scores are inconsistent across trials, reduce rubric ambiguity by adding concrete examples of passing and failing behavior.