maplibre-running-evals
The step-by-step procedure for running this repo's Promptfoo evals with a generator and a judge over provider APIs. Use when starting a new skill and needing a baseline before writing content, when a run stalls or returns no data, when a provider rate-limits, or when deciding whether an edit needs a retest.
maplibre · 已有 AI 审核记录
安装与使用
运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。
npx skills add maplibre/maplibre-agent-skills --skill maplibre-running-evals技能说明
name: maplibre-running-evals description: The step-by-step procedure for running this repo's Promptfoo evals with a generator and a judge over provider APIs. Use when starting a new skill and needing a baseline before writing content, when a run stalls or returns no data, when a provider rate-limits, or when deciding whether an edit needs a retest. metadata: status: process internal: true
Running Promptfoo Evals
Every eval here runs two roles over provider APIs: a generator, the model under test, and a judge, the model that grades each llm-rubric assertion. evals/README.md covers what to test and how to write prompts and rubrics; this is the procedure for running them. Work the phases in order — do not start at phase 5 because content already exists.
1. Set up and budget the run
Budget before anything else. These providers are used on free tiers, and a run you cannot finish teaches nothing.
- Confirm a generator key and a judge key are set — presence only, never the value:
echo "generator: ${GROQ_API_KEY:+set}"andecho "judge: ${GOOGLE_API_KEY:+set}". If either is missing, stop and report; do not extrapolate a result from a run you did not make. - Both roles are pinned: the generator in
evals/prompts/lib/providers.yaml, the judge inpackage.json'seval:gradedscript. Use the pinned pair by default. The one hard requirement is that they are different models — never grade a generator with itself — and any result you report names both. - Send every recorded run through
npm run eval:graded, the only place--grader,--delay, and-jare set. Do not hand-type them, and do not copy a value out of an older results doc: that value documents that run, not this one. Do not pass--providerseither: it replaces the pinned generator with a bare one, so the run is no longer pinned and its results are mislabeled — evals/README.md, "Writing eval prompts" has the detail.--providers echois the probe case: it returns the rendered prompt in place of a generator answer, so you see exactly what the generator would be sent — but everyllm-rubricstill calls the judge, so even a probe spends judge calls. - Learn the provider's limits before launching: read its rate-limit documentation for the pinned model's tier, then make one minimal request to the endpoint with response headers shown. On an OpenAI-compatible endpoint the
x-ratelimit-*headers carry the per-minute token window (limit, remaining, reset) and the per-day request count; a daily token cap has no header of its own, so it surfaces only when a429arrives —retry-aftergives the wait, and the body names which limit tripped. Header names and what they count are provider-specific and change, so confirm them against the provider's own page.
# Example against the current generator's OpenAI-compatible endpoint; the URL and model id come from providers.yaml.
curl -s -X POST https://api.groq.com/openai/v1/chat/completions \
-H "Authorization: Bearer $GROQ_API_KEY" -H "Content-Type: application/json" \
--data '{"model":"openai/gpt-oss-120b","messages":[{"role":"user","content":"hi"}],"max_tokens":1}' \
-D - | grep -iE 'ratelimit|retry-after|"error"'
- Do the arithmetic before launching. Tokens ≈ bytes / 4, so
wc -c skills/<skill>/SKILL.mdgives the injection cost per test; a with-skill run costs roughly tests × (that + the generator's configuredmax_tokens), plus one judge call per rubric, against whatever the day has left. A baseline run drops the injection. If the total does not fit the remaining budget, do not launch it whole. - Run sequentially, one config per command.
-j 1is pinned, and never run two graded evals at once — across terminals or across teammates. Limits are per account, not per process, so a429on either role kills both runs; chain them instead, leaving a full per-minute window between:<run one> && sleep 60 && <run two>. Each config already wireslib/skill-prompt.mjsandlib/providers.yaml, so there is nothing to assemble by hand; if you do write a Promptfoo command yourself, it still names exactly one--config— given several, Promptfoo merges theirdefaultTestblocks and the lastskillFilewins. --delaypaces requests; it does not create budget. A rate-limit failure that survives correct pacing with no concurrency is exhausted capacity — stop, take the reset time from the direct call above, and do not loop.- When the remaining window or day is smaller than the run, run fewer tests per invocation — never a larger delay.
--filter-pattern '<regex against the test description>'runs only the matching tests (quote it;npm runpasses it through a shell), and it is also how you probe one test of a run type before committing the whole config.--filter-failing <the run's .json output>re-runs only what failed or errored — phase 8 writes that file.
2. Write the tests before the content
- Copy
evals/prompts/TEMPLATE.yamltoevals/prompts/<skill>.yamland write four tests — explicit, implicit, anti-pattern, negative — before the skill has any content. evals/README.md, "Writing eval prompts" is the how; this skill is the running procedure.
3. Run the baseline, with the skill withheld
npm run eval:graded -- --config evals/prompts/<skill>.yaml --var injectSkill=false
- An explicit, implicit, or anti-pattern test that passes at baseline has no discriminating power. Redesign it. Never keep it as evidence, and never delete a test to