跳至正文
OpenAgentSkill
浏览已开放 · 目录记录日期 2026-11-21 · 自动更新和账户服务暂时关闭。
安全

audit-ml-pipeline

Owns the `audit/` folder: one `# %%` (jupytext percent) Python file per experiment, aligned 1:1 with `experiments/NN_<short_name>.py` and `journal/NN_<short_name>.md`, that loads the experiment's skore report **read-only** and uses bare-last-expression cells whose `__repr__` carries the audit's signal. The agent executes the audit file via the bundled in-process runner (`audit-ml-pipeline/scripts/run_cells.py` — IPython `InteractiveShell.run_cell`), which streams a markdown digest of each cell's stdout + last-expression repr to stdout (optionally also to a file). The digest fuels narrative work (the `JOURNAL.md` Status + History update, follow-up questions about a past experiment, cross-experiment comparison). Stops at "audit/NN_*.py is placed, executed, and the digest is available." Never calls `skore.evaluate(...)` or `project.put(...)`. TRIGGER — any of: - `iterate-ml-experiment` § 4 record-outcome — audit is dispatched FIRST (replaces scratch probes for metric extraction). - The us

probabl-ai · 已有 AI 审核记录

安装与使用

运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。

npx skills add probabl-ai/skills --skill audit-ml-pipeline

技能说明


name: audit-ml-pipeline description: > Owns the audit/ folder: one # %% (jupytext percent) Python file per experiment, aligned 1:1 with experiments/NN_<short_name>.py and journal/NN_<short_name>.md, that loads the experiment's skore report read-only and uses bare-last-expression cells whose __repr__ carries the audit's signal. The agent executes the audit file via the bundled in-process runner (audit-ml-pipeline/scripts/run_cells.py — IPython InteractiveShell.run_cell), which streams a markdown digest of each cell's stdout + last-expression repr to stdout (optionally also to a file). The digest fuels narrative work (the JOURNAL.md Status + History update, follow-up questions about a past experiment, cross-experiment comparison). Stops at "audit/NN_*.py is placed, executed, and the digest is available." Never calls skore.evaluate(...) or project.put(...).

TRIGGER — any of:

  • iterate-ml-experiment § 4 record-outcome — audit is dispatched FIRST (replaces scratch probes for metric extraction).
  • The user asks "audit experiment 02", "show me what 03 looks like", "re-audit 04 against the new report".
  • An experiment was re-run (same put() key overwritten) and the matching audit file needs re-execution.
  • The user wants a human-readable narrative of a past experiment without firing the full iterate-from-skore flow.

SKIP when: the design note isn't approved yet (route to iterate-ml-experiment); the experiment hasn't been run (no report on disk); the agent feature isn't installed (delegate to python-env-manager § "Agent feature"); the user is mining the report to source the next experiment (iterate-from-skore); the user wants to explore the raw dataset rather than a finished run's skore report (explore-ml-data — audit reads a report, not the data).

HOW TO USE: confirm the four-way stem pairing exists (journal/NN_*.md approved + experiments/NN_*.py exists + smoke test passed + report under that key in the Project), then place audit/NN_<short_name>.py from templates/audit.py, substituting the package name + the literal Project init block copied from experiments/<stem>.py. Execute via the bundled runner: pixi run -e agent python .agents/skills/audit-ml-pipeline/scripts/run_cells.py audit/<stem>.py. Read the Stop conditions and emit the Pre-flight checklist before any write or shell command. Always invoke python-api for skore symbol signatures — never write them from memory.

Audit ML Pipeline

Per-experiment, human-readable, agent-executable narrative of a skore report — produced by executing a bare-expression # %% file and reading the digest. Read-only against the skore Project.

Next-step pointers

Came here from…After audit, next is…
iterate-ml-experiment § 4 record-outcome→ Read audit digest, fill Status block + JOURNAL row
User free-text ("audit 02", "re-audit 04")→ Surface metrics to the user; no further dispatch
Re-run of an existing experiment→ Re-execute the existing audit file; surface diff if metrics changed

The audit is dispatched FIRST in § 4, before any scratch probes. The digest carries the checks summary and the metrics summary — it replaces ad-hoc scratch/<ts>_inspect_*.py files for the metric extraction step.

Where things live — visual map

PathDurabilityWho writes itWhat it holds
audit/<NN>_<short_name>.pyDurable (in git)This skill, once per experimentThe bare-expression cells. Source of truth. Can be opened as a notebook in JupyterLab / VS Code for the rich HTML view
scratch/audit/<stem>/audit.mdEphemeral (gitignored), optionalrun_cells.py when given a 2nd argPer-cell markdown digest: source + stdout + last-expression repr. Same content as stdout
Stdout from run_cells.pyCaptured by the bash toolrun_cells.py (always)Streamed digest — the agent reads this directly from the tool output

Mnemonic: audit/ is source (in git); scratch/audit/ and stdout are output. Never put the source .py under scratch/audit/. Never commit anything under scratch/audit/.

Read-only contract

The central rule. Surfaced as the first Stop condition below.

Allowed in audit/<stem>.py:

  • skore.Project(...) — open the project this experiment wrote to.
  • project.summarize() — list (key, id) pairs.
  • project.get(id) — load a specific report by id.
  • Every report.* accessor.
  • Imports from <pkg> (read-only inspection).

Forbidden in audit/<stem>.py:

  • skore.evaluate(...) — duplicates the report under the same key and pollutes summarize().
  • project.put(...) — same.
  • Writes outside scratch/audit/<stem>/ — no data/ writes, no reports/ writes, no edits to src/<pkg>/. The audit is a viewer.
  • Mutation of the loaded report that survives the cell (e.g. monkey-patching skore symbols).

The runner renders every cell's source + last-expression repr + stdout to the digest. A forbidden call surfaces in the digest (as a put row in a later summarize() cell, or as a **error:** section). The contract is visible, not invisible.

Sibling read-only consumers (different output shapes, same discipline): scratch/<ts>_*.py probes, iterate-from-skore's Backlog enrichment walk. See evaluate-ml-pipeline § Stop conditions for the three-consumer rule.

Stop conditions — read before anything else

  • Read-only against the skore Project. See § Read-only contract. Never skore.evaluate(...) or project.put(...) in an audit file.
  • project.get(...) is by id, not key. For hub mode, read the id from the URL printed by project.put(): https://…/<workspace>/<project>/<type-plural>/<N> → id is skore:report:<type-singular>:<N> (URL segment is plural; id uses the singular — drop the trailing s, e.g. cross-validations → `cr
阅读完整来源 ↗