Skip to content
OpenAgentSkill
Browsing is open · Catalog captured 2026-11-21 · Automatic updates and account services are paused.
Development

evaluate-ml-pipeline

Methodology for evaluating a single sklearn-compatible learner (in particular, the `SkrubLearner` produced by `build-ml-pipeline`). Owns: which entry point to call (`skore.evaluate` first, the explicit report classes when needed), which cross-validator to pick from scikit-learn's catalogue, how to consume the structural metadata (`groups`, `times`, …) attached at build time via `.skb.mark_as_X(split_kwargs=...)`. Stops at "what does the report say". Defaults (metrics, plots) come from skore; only override on explicit user request. TRIGGER when: code calls `cross_val_score`, `cross_validate`, `classification_report`, or any handwritten metric print (`print(mean_squared_error(...))`); code calls `.skb.cross_validate(...)` (route through skore for richer output); user asks how to score, evaluate, or compare a single learner; user asks how to pick a cross-validator; user wants to see a report / metrics / diagnostic plots for a fitted learner. SKIP when: declaring the pipeline (use `build-m

probabl-ai · Static checks on file

Installation

Read the source and dependencies before running a command. Copying does not install or execute anything.

npx skills add probabl-ai/skills --skill evaluate-ml-pipeline

About this skill


name: evaluate-ml-pipeline description: > Methodology for evaluating a single sklearn-compatible learner (in particular, the SkrubLearner produced by build-ml-pipeline). Owns: which entry point to call (skore.evaluate first, the explicit report classes when needed), which cross-validator to pick from scikit-learn's catalogue, how to consume the structural metadata (groups, times, …) attached at build time via .skb.mark_as_X(split_kwargs=...). Stops at "what does the report say". Defaults (metrics, plots) come from skore; only override on explicit user request.

TRIGGER when: code calls cross_val_score, cross_validate, classification_report, or any handwritten metric print (print(mean_squared_error(...))); code calls .skb.cross_validate(...) (route through skore for richer output); user asks how to score, evaluate, or compare a single learner; user asks how to pick a cross-validator; user wants to see a report / metrics / diagnostic plots for a fitted learner.

SKIP when: declaring the pipeline (use build-ml-pipeline); hyperparameter / model search (separate skill); fitting, persisting, or serving the final model; tracking or comparing experiments across multiple runs over time (separate skill).

HOW TO USE: invoke before any evaluation call. First, read the "Stop conditions" block at the top of the body and emit the Pre-flight checklist as visible text in your response — both are mandatory before any evaluation code is written. The structural facts about the data (group keys, time ordering) should already be encoded at the X marker via split_kwargs — if they aren't and you can't tell from the data, return to build-ml-pipeline and ask the user. For symbol-level lookups, defer to python-api (skore symbols) and python-api (splitters); don't guess names from memory.

Evaluate ML Pipeline

Pick the entry point, pick the cross-validator, route the metadata, read the report. The pipeline declaration is out of scope (see build-ml-pipeline).

Stop conditions — read before anything else

  • Missing dependency. If import skore raises in this project's env, STOP. Invoke python-env-manager to detect the manager and produce the right install command (the project may not use pixi); surface the command to the user and wait for confirmation. Do not drop back to cross_val_score, cross_validate, classification_report, or hand-rolled metric prints — that silently rewrites this skill out of the project. See data-science-python-stack § "Missing dependency".
  • Symbol from memory is forbidden. Any skore entry point (evaluate, EstimatorReport, CrossValidationReport, ComparisonReport) and any sklearn splitter name must come from a Skill(python-api) or Skill(python-api) call in this turn. "I remember KFold(n_splits=5)" is not acceptable.
  • Splitter choice is data-driven, not default-driven (G-CV-SPLITTER). This is the G-CV-SPLITTER gate — owned by this skill, fired during iterate-ml-experiment § 3 (the build → evaluate → test chain, after the design note is approved at G-DESIGN), before src/<pkg>/evaluate.py is written. The splitter is NOT pre-committed in the design note. Pick from the split_kwargs content at the X marker via the table in rule 3 — never reach for KFold(5) or StratifiedKFold out of habit. If split_kwargs is empty and you cannot rule out group / temporal structure, return to build-ml-pipeline and ask before defaulting.
  • No Stratified* for class imbalance. It compresses across-fold variance and produces over-confident error bars. Imbalance does not change the splitter choice.
  • CV is necessary but not sufficient for any pipeline with history-dependent features. skore.evaluate(...) materializes the graph once with one env-dict and splits indices — it never exercises a different env-dict at predict time, which is exactly the binding shape production faces. A pipeline that loads-then-features-then-splits passes CV trivially and still silently drops cold-start rows when handed a fresh learner.predict(env₂). The structural check that catches this is the smoke test owned by smoke-test-ml-pipeline — required alongside CV for any pipeline that has a backward shift, lag, rolling window, target shift, or join with side history. If you produce a CV report and the pipeline has any such step, the matching tests/smoke/test_NN_<short_name>.py must also pass before the experiment can flip to done (enforced by iterate-ml-experiment § 4).
  • All Python execution goes to scratch/. Every Python command — version checks, signature lookups, walking the skore report's metrics accessors, extracting per-fold values, sanity-checking the splitter's fold geometry, multi-symbol inspect.signature(...) on skore / sklearn classes — lands in scratch/<YYYY-MM-DD>_<HHMMSS>_<short>.py and runs via pixi run python scratch/<ts>_<short>.py. Inline pixi run python -c "..." is forbidden regardless of length (see python-api § Stop conditions). The previous "2-line inline cap" is removed.
  • Don't filter warnings. No warnings.filterwarnings(...) around skore.evaluate(...) or the CV splitter unless the user explicitly asks. See python-code-style § Stop conditions.
  • skore.evaluate(...) and project.put(...) live only in experiments/NN_*.py. The experiment script is the sole producer of a report in the workspace's skore Project. Re-running evaluate from a scratch/ probe, an audit/ file, a notebook, or a one-off Python file in src/ duplicates the report under the same key and pollutes project.summarize() — the cross-experiment metrics view the audit digest draws from. Two read-only consumers of the Project share the same summarize() → get(id) → report.* discipline:
Read the complete source ↗