evaluate-ml-pipeline
Methodology for evaluating a single sklearn-compatible learner (in particular, the `SkrubLearner` produced by `build-ml-pipeline`). Owns: which entry point to call (`skore.evaluate` first, the explicit report classes when needed), which cross-validator to pick from scikit-learn's catalogue, how to consume the structural metadata (`groups`, `times`, …) attached at build time via `.skb.mark_as_X(split_kwargs=...)`. Stops at "what does the report say". Defaults (metrics, plots) come from skore; only override on explicit user request. TRIGGER when: code calls `cross_val_score`, `cross_validate`, `classification_report`, or any handwritten metric print (`print(mean_squared_error(...))`); code calls `.skb.cross_validate(...)` (route through skore for richer output); user asks how to score, evaluate, or compare a single learner; user asks how to pick a cross-validator; user wants to see a report / metrics / diagnostic plots for a fitted learner. SKIP when: declaring the pipeline (use `build-m
probabl-ai · 已有静态检查记录
安装与使用
运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。
npx skills add probabl-ai/skills --skill evaluate-ml-pipeline技能说明
name: evaluate-ml-pipeline
description: >
Methodology for evaluating a single sklearn-compatible learner (in
particular, the SkrubLearner produced by build-ml-pipeline).
Owns: which entry point to call (skore.evaluate first, the
explicit report classes when needed), which cross-validator to pick
from scikit-learn's catalogue, how to consume the structural
metadata (groups, times, …) attached at build time via
.skb.mark_as_X(split_kwargs=...). Stops at "what does the report
say". Defaults (metrics, plots) come from skore; only override on
explicit user request.
TRIGGER when: code calls cross_val_score, cross_validate,
classification_report, or any handwritten metric print
(print(mean_squared_error(...))); code calls
.skb.cross_validate(...) (route through skore for richer output);
user asks how to score, evaluate, or compare a single learner;
user asks how to pick a cross-validator; user wants to see a
report / metrics / diagnostic plots for a fitted learner.
SKIP when: declaring the pipeline (use build-ml-pipeline);
hyperparameter / model search (separate skill); fitting,
persisting, or serving the final model; tracking or comparing
experiments across multiple runs over time (separate skill).
HOW TO USE: invoke before any evaluation call. First, read the
"Stop conditions" block at the top of the body and emit the
Pre-flight checklist as visible text in your response — both are
mandatory before any evaluation code is written. The structural
facts about the data (group keys, time ordering) should already be
encoded at the X marker via split_kwargs — if they aren't and you
can't tell from the data, return to build-ml-pipeline and ask the
user. For symbol-level lookups, defer to python-api (skore
symbols) and python-api (splitters); don't guess names from
memory.
Evaluate ML Pipeline
Pick the entry point, pick the cross-validator, route the metadata,
read the report. The pipeline declaration is out of scope (see
build-ml-pipeline).
Stop conditions — read before anything else
- Missing dependency. If
import skoreraises in this project's env, STOP. Invokepython-env-managerto detect the manager and produce the right install command (the project may not use pixi); surface the command to the user and wait for confirmation. Do not drop back tocross_val_score,cross_validate,classification_report, or hand-rolled metric prints — that silently rewrites this skill out of the project. Seedata-science-python-stack§ "Missing dependency". - Symbol from memory is forbidden. Any
skoreentry point (evaluate,EstimatorReport,CrossValidationReport,ComparisonReport) and any sklearn splitter name must come from aSkill(python-api)orSkill(python-api)call in this turn. "I rememberKFold(n_splits=5)" is not acceptable. - Splitter choice is data-driven, not default-driven
(
G-CV-SPLITTER). This is the G-CV-SPLITTER gate — owned by this skill, fired duringiterate-ml-experiment§ 3 (the build → evaluate → test chain, after the design note is approved at G-DESIGN), beforesrc/<pkg>/evaluate.pyis written. The splitter is NOT pre-committed in the design note. Pick from thesplit_kwargscontent at the X marker via the table in rule 3 — never reach forKFold(5)orStratifiedKFoldout of habit. Ifsplit_kwargsis empty and you cannot rule out group / temporal structure, return tobuild-ml-pipelineand ask before defaulting. - No
Stratified*for class imbalance. It compresses across-fold variance and produces over-confident error bars. Imbalance does not change the splitter choice. - CV is necessary but not sufficient for any pipeline with
history-dependent features.
skore.evaluate(...)materializes the graph once with one env-dict and splits indices — it never exercises a different env-dict at predict time, which is exactly the binding shape production faces. A pipeline that loads-then-features-then-splits passes CV trivially and still silently drops cold-start rows when handed a freshlearner.predict(env₂). The structural check that catches this is the smoke test owned bysmoke-test-ml-pipeline— required alongside CV for any pipeline that has a backward shift, lag, rolling window, target shift, or join with side history. If you produce a CV report and the pipeline has any such step, the matchingtests/smoke/test_NN_<short_name>.pymust also pass before the experiment can flip todone(enforced byiterate-ml-experiment§ 4). - All Python execution goes to
scratch/. Every Python command — version checks, signature lookups, walking the skore report's metrics accessors, extracting per-fold values, sanity-checking the splitter's fold geometry, multi-symbolinspect.signature(...)on skore / sklearn classes — lands inscratch/<YYYY-MM-DD>_<HHMMSS>_<short>.pyand runs viapixi run python scratch/<ts>_<short>.py. Inlinepixi run python -c "..."is forbidden regardless of length (seepython-api§ Stop conditions). The previous "2-line inline cap" is removed. - Don't filter warnings. No
warnings.filterwarnings(...)aroundskore.evaluate(...)or the CV splitter unless the user explicitly asks. Seepython-code-style§ Stop conditions. skore.evaluate(...)andproject.put(...)live only inexperiments/NN_*.py. The experiment script is the sole producer of a report in the workspace's skore Project. Re-runningevaluatefrom ascratch/probe, anaudit/file, a notebook, or a one-off Python file insrc/duplicates the report under the samekeyand pollutesproject.summarize()— the cross-experiment metrics view the audit digest draws from. Two read-only consumers of the Project share the samesummarize()→get(id)→report.*discipline: