smoke-test-ml-pipeline
Owns the smoke test contract for an ML experiment: a small, diagnostic-by-construction pytest that fits the experiment's learner on a portion of the real `data/` source and predicts on a *disjoint* portion that deliberately carries **no pre-history buffer**. The assertion is structural — the number of predictions must equal the number of rows in the predict grid. A pipeline that loads-then-features-then-splits will silently drop the cold-start rows of the predict slice and the test will fail with a row-count mismatch; a pipeline that marks X early and references upstream history nodes from feature steps will pass trivially. The smoke test is the executable proof of the X-marker placement rule from `build-ml-pipeline`. TRIGGER when: `test-ml-pipeline` has dispatched here to write the smoke test for an approved experiment; `pytest tests/smoke/` is failing on row count; the user asks "why is the smoke test failing?"; a pipeline edit in `build-ml-pipeline` needs an executable proof; an exp
probabl-ai · AI review on file
Installation
Read the source and dependencies before running a command. Copying does not install or execute anything.
npx skills add probabl-ai/skills --skill smoke-test-ml-pipelineAbout this skill
name: smoke-test-ml-pipeline
description: >
Owns the smoke test contract for an ML experiment: a small,
diagnostic-by-construction pytest that fits the experiment's
learner on a portion of the real data/ source and predicts
on a disjoint portion that deliberately carries no
pre-history buffer. The assertion is structural — the number
of predictions must equal the number of rows in the predict
grid. A pipeline that loads-then-features-then-splits will
silently drop the cold-start rows of the predict slice and the
test will fail with a row-count mismatch; a pipeline that marks
X early and references upstream history nodes from feature
steps will pass trivially. The smoke test is the executable
proof of the X-marker placement rule from build-ml-pipeline.
TRIGGER when: test-ml-pipeline has dispatched here to write
the smoke test for an approved experiment; pytest tests/smoke/
is failing on row count; the user asks "why is the smoke test
failing?"; a pipeline edit in build-ml-pipeline needs an
executable proof; an experiment script changes the pipeline
shape and the matching smoke test needs revisiting.
SKIP when: the design note does not exist or is not yet
approved (route to iterate-ml-experiment); the user is asking
about a regression test or schema invariant (route to
regression-test-ml-pipeline /
distribution-test-ml-pipeline once those exist); the question
is the interpretation of CV metrics, not predict-time
correctness (route to evaluate-ml-pipeline).
HOW TO USE: read the matching experiment's journal/NN_*.md and
experiments/NN_*.py first to understand the pipeline's source
binding (what env-dict keys does build_learner expect?). Then
construct two env-dicts from the real data/ source — a
train env and a predict env — such that the predict env carries
only the rows we want predictions for and no pre-history
buffer. The hard assertion is that the prediction count
matches the predict-env row count exactly. The soft assertion
is that the smoke set's MAE is within 3 × CV_mean (or the
task-appropriate analogue). Do not write the design note
or run CV — that's other skills' job.
Smoke Test ML Pipeline
The minimal pytest that catches the "load → featurize → split" anti-pattern at iteration time, before it reaches production.
Stop conditions — read before anything else
- No smoke test without an approved design note + script. The pairing
rule from
test-ml-pipelineis hard:tests/smoke/test_NN_<short_name>.pyexists only whenjournal/NN_<short_name>.mdis at leastapprovedandexperiments/NN_<short_name>.pyexists with the matching stem. - Symbol from memory is forbidden. Any skrub /
scikit-learn name you write in the smoke test must come from a
Skill(python-api)/Skill(python-api)call in this turn. The smoke test is a small file but it imports the predicting-package API surface; the same memory-forbidden rule applies. - Don't shrink the assertion. The hard assertion is exact row-count equality. Not "approximately equal", not "at least 80% of expected rows". A row-count mismatch is the failure mode the smoke test exists to catch. Loosening the assertion silently reintroduces the bug.
- Don't synthesize the fixture. The smoke test reads the real
data/source. Synthetic fixtures look fine but skip the loaders that actually break in production. - No wrappers, no NaN-handling, no
eval_modehacks. If the smoke test only passes after wrapping the predictor or conditioning oneval_mode, the pipeline is wrong. Route back tobuild-ml-pipelineand fix the X-marker placement. Wrappers paper over the failure mode; they don't solve it. - The smoke test uses only the predicting package's API.
For a
SkrubLearnerproduced bybuild-ml-pipelinethat means skrub'sfit/predict/ (optionallyscore) plussklearn.metricsfor any metric the soft assertion uses. Do not importskore(or any other tracking / reporting library) in the test file. The smoke test must be runnable in any environment that canimport skrub+import sklearn— the skore Project is a side artifact, not a test dependency. Soft-assertion baselines (CV-mean MAE, etc.) are hardcoded from the design note's Status.headline with a comment pointing to the design note; update by hand when the experiment's headline number changes. - Don't filter warnings. No
@pytest.mark.filterwarnings(...), nowarnings.filterwarnings(...)in the test body, nofilterwarnings = [...]inpytest.ini/pyproject.toml— unless the user explicitly asks. Seepython-code-style§ Stop conditions.
Pre-flight — emit this checklist as visible text before any test code
Pre-flight (smoke-test-ml-pipeline):
- [ ] Tier 1 mandatory libs importable: pytest + sklearn + skrub
(per `data-science-python-stack` § "Tier 1"). **Not skore** —
see the Stop conditions; the smoke test is intentionally
portable to any skrub-capable environment
- [ ] Skill(python-api) consulted for skrub / sklearn symbols used in
the test: <symbols, or "none">
Evidence: Read scratch/api/<lib>/<version>/<topic>.md (this turn)
| Write scratch/api/<lib>/<version>/<topic>.md (this turn)
| "n/a — test only uses symbols already present in
src/<pkg>/ (build_learner / load_training_table / etc.)"
"Read python-api SKILL.md" alone is NOT evidence.
- [ ] `journal/NN_<short_name>.md` read this turn (frozen sections:
Question, Method) so the test asserts what the experiment claims
- [ ] `experiments/NN_<short_name>.py` skimmed this turn for the env-dict
keys `build_learner` consumes (`data_dir` / `start` + `end` /
`raw_frame` / etc.)
- [ ] `src/<pkg>/data.py` skimmed this turn for the loader signature
(so the
Read the complete source ↗