跳至正文
OpenAgentSkill
浏览已开放 · 目录记录日期 2026-11-21 · 自动更新和账户服务暂时关闭。
其他技能

maplibre-running-evals

The step-by-step procedure for running this repo's Promptfoo evals with a generator and a judge over provider APIs. Use when starting a new skill and needing a baseline before writing content, when a run stalls or returns no data, when a provider rate-limits, or when deciding whether an edit needs a retest.

maplibre · 已有 AI 审核记录

安装与使用

运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。

npx skills add maplibre/maplibre-agent-skills --skill maplibre-running-evals

技能说明


name: maplibre-running-evals description: The step-by-step procedure for running this repo's Promptfoo evals with a generator and a judge over provider APIs. Use when starting a new skill and needing a baseline before writing content, when a run stalls or returns no data, when a provider rate-limits, or when deciding whether an edit needs a retest. metadata: status: process internal: true

Running Promptfoo Evals

Every eval here runs two roles over provider APIs: a generator, the model under test, and a judge, the model that grades each llm-rubric assertion. evals/README.md covers what to test and how to write prompts and rubrics; this is the procedure for running them. Work the phases in order — do not start at phase 5 because content already exists.

1. Set up and budget the run

Budget before anything else. These providers are used on free tiers, and a run you cannot finish teaches nothing.

  • Confirm a generator key and a judge key are set — presence only, never the value: echo "generator: ${GROQ_API_KEY:+set}" and echo "judge: ${GOOGLE_API_KEY:+set}". If either is missing, stop and report; do not extrapolate a result from a run you did not make.
  • Both roles are pinned: the generator in evals/prompts/lib/providers.yaml, the judge in package.json's eval:graded script. Use the pinned pair by default. The one hard requirement is that they are different models — never grade a generator with itself — and any result you report names both.
  • Send every recorded run through npm run eval:graded, the only place --grader, --delay, and -j are set. Do not hand-type them, and do not copy a value out of an older results doc: that value documents that run, not this one. Do not pass --providers either: it replaces the pinned generator with a bare one, so the run is no longer pinned and its results are mislabeled — evals/README.md, "Writing eval prompts" has the detail. --providers echo is the probe case: it returns the rendered prompt in place of a generator answer, so you see exactly what the generator would be sent — but every llm-rubric still calls the judge, so even a probe spends judge calls.
  • Learn the provider's limits before launching: read its rate-limit documentation for the pinned model's tier, then make one minimal request to the endpoint with response headers shown. On an OpenAI-compatible endpoint the x-ratelimit-* headers carry the per-minute token window (limit, remaining, reset) and the per-day request count; a daily token cap has no header of its own, so it surfaces only when a 429 arrives — retry-after gives the wait, and the body names which limit tripped. Header names and what they count are provider-specific and change, so confirm them against the provider's own page.
# Example against the current generator's OpenAI-compatible endpoint; the URL and model id come from providers.yaml.
curl -s -X POST https://api.groq.com/openai/v1/chat/completions \
  -H "Authorization: Bearer $GROQ_API_KEY" -H "Content-Type: application/json" \
  --data '{"model":"openai/gpt-oss-120b","messages":[{"role":"user","content":"hi"}],"max_tokens":1}' \
  -D - | grep -iE 'ratelimit|retry-after|"error"'
  • Do the arithmetic before launching. Tokens ≈ bytes / 4, so wc -c skills/<skill>/SKILL.md gives the injection cost per test; a with-skill run costs roughly tests × (that + the generator's configured max_tokens), plus one judge call per rubric, against whatever the day has left. A baseline run drops the injection. If the total does not fit the remaining budget, do not launch it whole.
  • Run sequentially, one config per command. -j 1 is pinned, and never run two graded evals at once — across terminals or across teammates. Limits are per account, not per process, so a 429 on either role kills both runs; chain them instead, leaving a full per-minute window between: <run one> && sleep 60 && <run two>. Each config already wires lib/skill-prompt.mjs and lib/providers.yaml, so there is nothing to assemble by hand; if you do write a Promptfoo command yourself, it still names exactly one --config — given several, Promptfoo merges their defaultTest blocks and the last skillFile wins.
  • --delay paces requests; it does not create budget. A rate-limit failure that survives correct pacing with no concurrency is exhausted capacity — stop, take the reset time from the direct call above, and do not loop.
  • When the remaining window or day is smaller than the run, run fewer tests per invocation — never a larger delay. --filter-pattern '<regex against the test description>' runs only the matching tests (quote it; npm run passes it through a shell), and it is also how you probe one test of a run type before committing the whole config. --filter-failing <the run's .json output> re-runs only what failed or errored — phase 8 writes that file.

2. Write the tests before the content

3. Run the baseline, with the skill withheld

npm run eval:graded -- --config evals/prompts/<skill>.yaml --var injectSkill=false

  • An explicit, implicit, or anti-pattern test that passes at baseline has no discriminating power. Redesign it. Never keep it as evidence, and never delete a test to
阅读完整来源 ↗