audit-ml-pipeline
Owns the `audit/` folder: one `# %%` (jupytext percent) Python file per experiment, aligned 1:1 with `experiments/NN_<short_name>.py` and `journal/NN_<short_name>.md`, that loads the experiment's skore report **read-only** and uses bare-last-expression cells whose `__repr__` carries the audit's signal. The agent executes the audit file via the bundled in-process runner (`audit-ml-pipeline/scripts/run_cells.py` — IPython `InteractiveShell.run_cell`), which streams a markdown digest of each cell's stdout + last-expression repr to stdout (optionally also to a file). The digest fuels narrative work (the `JOURNAL.md` Status + History update, follow-up questions about a past experiment, cross-experiment comparison). Stops at "audit/NN_*.py is placed, executed, and the digest is available." Never calls `skore.evaluate(...)` or `project.put(...)`. TRIGGER — any of: - `iterate-ml-experiment` § 4 record-outcome — audit is dispatched FIRST (replaces scratch probes for metric extraction). - The us
probabl-ai · AI review on file
Installation
Read the source and dependencies before running a command. Copying does not install or execute anything.
npx skills add probabl-ai/skills --skill audit-ml-pipelineAbout this skill
name: audit-ml-pipeline
description: >
Owns the audit/ folder: one # %% (jupytext percent) Python file
per experiment, aligned 1:1 with experiments/NN_<short_name>.py and
journal/NN_<short_name>.md, that loads the experiment's skore
report read-only and uses bare-last-expression cells whose
__repr__ carries the audit's signal. The agent executes the audit
file via the bundled in-process runner
(audit-ml-pipeline/scripts/run_cells.py — IPython
InteractiveShell.run_cell), which streams a markdown digest of
each cell's stdout + last-expression repr to stdout (optionally also
to a file). The digest fuels narrative work (the JOURNAL.md
Status + History update, follow-up questions about a past
experiment, cross-experiment comparison). Stops at "audit/NN_*.py
is placed, executed, and the digest is available." Never calls
skore.evaluate(...) or project.put(...).
TRIGGER — any of:
iterate-ml-experiment§ 4 record-outcome — audit is dispatched FIRST (replaces scratch probes for metric extraction).- The user asks "audit experiment 02", "show me what 03 looks like", "re-audit 04 against the new report".
- An experiment was re-run (same
put()key overwritten) and the matching audit file needs re-execution. - The user wants a human-readable narrative of a past experiment
without firing the full
iterate-from-skoreflow.
SKIP when: the design note isn't approved yet (route to
iterate-ml-experiment); the experiment hasn't been run (no report
on disk); the agent feature isn't installed (delegate to
python-env-manager § "Agent feature"); the user is mining the
report to source the next experiment (iterate-from-skore); the
user wants to explore the raw dataset rather than a finished
run's skore report (explore-ml-data — audit reads a report, not
the data).
HOW TO USE: confirm the four-way stem pairing exists (journal/NN_*.md
approved + experiments/NN_*.py exists + smoke test passed +
report under that key in the Project), then place
audit/NN_<short_name>.py from templates/audit.py, substituting
the package name + the literal Project init block copied from
experiments/<stem>.py. Execute via the bundled runner: pixi run -e agent python .agents/skills/audit-ml-pipeline/scripts/run_cells.py audit/<stem>.py. Read the Stop conditions and emit the Pre-flight
checklist before any write or shell command. Always invoke
python-api for skore symbol signatures — never write them from
memory.
Audit ML Pipeline
Per-experiment, human-readable, agent-executable narrative of a skore
report — produced by executing a bare-expression # %% file and
reading the digest. Read-only against the skore Project.
Next-step pointers
| Came here from… | After audit, next is… |
|---|---|
iterate-ml-experiment § 4 record-outcome | → Read audit digest, fill Status block + JOURNAL row |
| User free-text ("audit 02", "re-audit 04") | → Surface metrics to the user; no further dispatch |
| Re-run of an existing experiment | → Re-execute the existing audit file; surface diff if metrics changed |
The audit is dispatched FIRST in § 4, before any scratch probes.
The digest carries the checks summary and the metrics summary — it
replaces ad-hoc scratch/<ts>_inspect_*.py files for the metric
extraction step.
Where things live — visual map
| Path | Durability | Who writes it | What it holds |
|---|---|---|---|
audit/<NN>_<short_name>.py | Durable (in git) | This skill, once per experiment | The bare-expression cells. Source of truth. Can be opened as a notebook in JupyterLab / VS Code for the rich HTML view |
scratch/audit/<stem>/audit.md | Ephemeral (gitignored), optional | run_cells.py when given a 2nd arg | Per-cell markdown digest: source + stdout + last-expression repr. Same content as stdout |
Stdout from run_cells.py | Captured by the bash tool | run_cells.py (always) | Streamed digest — the agent reads this directly from the tool output |
Mnemonic: audit/ is source (in git); scratch/audit/ and
stdout are output. Never put the source .py under
scratch/audit/. Never commit anything under scratch/audit/.
Read-only contract
The central rule. Surfaced as the first Stop condition below.
Allowed in audit/<stem>.py:
skore.Project(...)— open the project this experiment wrote to.project.summarize()— list(key, id)pairs.project.get(id)— load a specific report by id.- Every
report.*accessor. - Imports from
<pkg>(read-only inspection).
Forbidden in audit/<stem>.py:
skore.evaluate(...)— duplicates the report under the same key and pollutessummarize().project.put(...)— same.- Writes outside
scratch/audit/<stem>/— nodata/writes, noreports/writes, no edits tosrc/<pkg>/. The audit is a viewer. - Mutation of the loaded
reportthat survives the cell (e.g. monkey-patching skore symbols).
The runner renders every cell's source + last-expression repr +
stdout to the digest. A forbidden call surfaces in the digest (as a
put row in a later summarize() cell, or as a **error:**
section). The contract is visible, not invisible.
Sibling read-only consumers (different output shapes, same
discipline): scratch/<ts>_*.py probes, iterate-from-skore's
Backlog enrichment walk. See evaluate-ml-pipeline § Stop
conditions for the three-consumer rule.
Stop conditions — read before anything else
- Read-only against the skore Project. See § Read-only contract.
Never
skore.evaluate(...)orproject.put(...)in an audit file. project.get(...)is by id, not key. For hub mode, read the id from the URL printed byproject.put():https://…/<workspace>/<project>/<type-plural>/<N>→ id isskore:report:<type-singular>:<N>(URL segment is plural; id uses the singular — drop the trailings, e.g.cross-validations→ `cr