audit-ml-pipeline

Dokumentasi sumber, bukan instruksi untuk situs ini. Periksa izin sebelum menjalankan perintah.

Pre-flight (audit-ml-pipeline):
- [ ] Experiment stem confirmed: <NN_short_name>
      Evidence: journal/NN_<short_name>.md exists AND state ≥ done
                | "n/a — user invoked re-audit on existing stem"
- [ ] Four-way pairing complete:
        journal/NN_<short_name>.md       — design note (state ≥ done)
        experiments/NN_<short_name>.py   — script
        tests/smoke/test_NN_<short_name>.py — smoke test (passing)
        audit/NN_<short_name>.py         — about to be written / refreshed
      Evidence: ls / Glob on each path
- [ ] Report present in skore Project under key=<NN_short_name>
      Evidence: scratch/<ts>_check_report.py probe ran
                project.summarize() this turn; row with
                key == "<NN_short_name>" appears.
                "Run finished, put() landed" is NOT sufficient.
- [ ] Agent feature available:
        `pixi run -e agent ipython -c "print(0)"` exit 0
        `pixi run -e agent pyright --version` exit 0
      Evidence: tool output of each
                | JOURNAL.md Status `agent feature: installed`
                Missing → STOP, delegate to python-env-manager G-AGENT-FEATURE
- [ ] python-api consulted for skore symbols used:
      Project, summarize, get, report.checks.summarize, report.metrics.summarize
      Evidence: Read scratch/api/skore/<version>/<topic>.md (this turn)
                | Write the same (this turn)
                | "n/a — cache hit, file already on disk + Read this turn"
- [ ] Template copy + substitution decided:
        <pkg> → package name from src/<pkg>/
        <NN>_<short_name> → experiment stem
        <SKORE_PROJECT_INIT> → literal block copi

name: audit-ml-pipeline
description: >
  Owns the `audit/` folder: one `# %%` (jupytext percent) Python file
  per experiment, aligned 1:1 with `experiments/NN_<short_name>.py` and
  `journal/NN_<short_name>.md`, that loads the experiment's skore
  report **read-only** and uses bare-last-expression cells whose
  `__repr__` carries the audit's signal.   The agent executes the audit
  file via the bundled in-process runner
  (`audit-ml-pipeline/scripts/run_cells.py` — IPython
  `InteractiveShell.run_cell`), which streams a markdown digest of
  each cell's stdout + last-expression repr to stdout (optionally also
  to a file). The digest fuels narrative work (the `JOURNAL.md`
  Status + History update, follow-up questions about a past
  experiment, cross-experiment comparison). Stops at "audit/NN_*.py
  is placed, executed, and the digest is available." Never calls
  `skore.evaluate(...)` or `project.put(...)`.

  TRIGGER — any of:
  - `iterate-ml-experiment` § 4 record-outcome — audit is dispatched
    FIRST (replaces scratch probes for metric extraction).
  - The user asks "audit experiment 02", "show me what 03 looks
    like", "re-audit 04 against the new report".
  - An experiment was re-run (same `put()` key overwritten) and the
    matching audit file needs re-execution.
  - The user wants a human-readable narrative of a past experiment
    without firing the full `iterate-from-skore` flow.

  SKIP when: the design note isn't approved yet (route to
  `iterate-ml-experiment`); the experiment hasn't been run (no report
  on disk); the agent feature isn't installed (delegate to
  `python-env-manager` § "Agent feature"); the user is mining the
  report to source the *next* experiment (`iterate-from-skore`); the
  user wants to explore the **raw dataset** rather than a finished
  run's skore report (`explore-ml-data` — audit reads a report, not
  the data).

  HOW TO USE: confirm the four-way stem pairing exists (`journal/NN_*.md`
  approved + `experiments/NN_*.py` exists + smoke test passed +
  report under that key in the Project), then place
  `audit/NN_<short_name>.py` from `templates/audit.py`, substituting
  the package name + the literal Project init block copied from
  `experiments/<stem>.py`.   Execute via the bundled runner: `pixi run
  -e agent python .agents/skills/audit-ml-pipeline/scripts/run_cells.py
  audit/<stem>.py`. **Read the Stop conditions and emit the Pre-flight
  checklist before any write or shell command.** Always invoke
  `python-api` for skore symbol signatures — never write them from
  memory.

---
name: audit-ml-pipeline
description: >
  Owns the `audit/` folder: one `# %%` (jupytext percent) Python file
  per experiment, aligned 1:1 with `experiments/NN_<short_name>.py` and
  `journal/NN_<short_name>.md`, that loads the experiment's skore
  report **read-only** and uses bare-last-expression cells whose
  `__repr__` carries the audit's signal.   The agent executes the audit
  file via the bundled in-process runner
  (`audit-ml-pipeline/scripts/run_cells.py` — IPython
  `InteractiveShell.run_cell`), which streams a markdown digest of
  each cell's stdout + last-expression repr to stdout (optionally also
  to a file). The digest fuels narrative work (the `JOURNAL.md`
  Status + History update, follow-up questions about a past
  experiment, cross-experiment comparison). Stops at "audit/NN_*.py
  is placed, executed, and the digest is available." Never calls
  `skore.evaluate(...)` or `project.put(...)`.

  TRIGGER — any of:
  - `iterate-ml-experiment` § 4 record-outcome — audit is dispatched
    FIRST (replaces scratch probes for metric extraction).
  - The user asks "audit experiment 02", "show me what 03 looks
    like", "re-audit 04 against the new report".
  - An experiment was re-run (same `put()` key overwritten) and the
    matching audit file needs re-execution.
  - The user wants a human-readable narrative of a past experiment
    without firing the full `iterate-from-skore` flow.

  SKIP when: the design note isn't approved yet (route to
  `iterate-ml-experiment`); the experiment hasn't been run (no report
  on disk); the agent feature isn't installed (delegate to
  `python-env-manager` § "Agent feature"); the user is mining the
  report to source the *next* experiment (`iterate-from-skore`); the
  user wants to explore the **raw dataset** rather than a finished
  run's skore report (`explore-ml-data` — audit reads a report, not
  the data).

  HOW TO USE: confirm the four-way stem pairing exists (`journal/NN_*.md`
  approved + `experiments/NN_*.py` exists + smoke test passed +
  report under that key in the Project), then place
  `audit/NN_<short_name>.py` from `templates/audit.py`, substituting
  the package name + the literal Project init block copied from
  `experiments/<stem>.py`.   Execute via the bundled runner: `pixi run
  -e agent python .agents/skills/audit-ml-pipeline/scripts/run_cells.py
  audit/<stem>.py`. **Read the Stop conditions and emit the Pre-flight
  checklist before any write or shell command.** Always invoke
  `python-api` for skore symbol signatures — never write them from
  memory.
---

# Audit ML Pipeline

Per-experiment, human-readable, agent-executable narrative of a skore
report — produced by **executing** a bare-expression `# %%` file and
reading the digest. Read-only against the skore Project.

## Next-step pointers

| Came here from… | After audit, next is… |
|---|---|
| `iterate-ml-experiment` § 4 record-outcome | → Read audit digest, fill Status block + JOURNAL row |
| User free-text ("audit 02", "re-audit 04") | → Surface metrics to the user; no further dispatch |
| Re-run of an existing experiment | → Re-execute the existing audit file; surface diff if metrics changed |

The audit is dispatched **FIRST** in § 4, before any scratch probes.
The digest carries the checks summary and the metrics summary — it
replaces ad-hoc `scratch/<ts>_inspect_*.py` files for the metric
extraction step.

## Where things live — visual map

| Path | Durability | Who writes it | What it holds |
|---|---|---|---|
| `audit/<NN>_<short_name>.py` | **Durable** (in git) | This skill, once per experiment | The bare-expression cells. Source of truth. Can be opened as a notebook in JupyterLab / VS Code for the rich HTML view |
| `scratch/audit/<stem>/audit.md` | Ephemeral (gitignored), optional | `run_cells.py` when given a 2nd arg | Per-cell markdown digest: source + stdout + last-expression `repr`. Same content as stdout |
| Stdout from `run_cells.py` | Captured by the bash tool | `run_cells.py` (always) | Streamed digest — the agent reads this directly from the tool output |

**Mnemonic:** `audit/` is *source* (in git); `scratch/audit/` and
stdout are *output*. Never put the source `.py` under
`scratch/audit/`. Never commit anything under `scratch/audit/`.

## Read-only contract

The central rule. Surfaced as the first Stop condition below.

**Allowed in `audit/<stem>.py`:**

- `skore.Project(...)` — open the project this experiment wrote to.
- `project.summarize()` — list `(key, id)` pairs.
- `project.get(id)` — load a specific report by id.
- Every `report.*` accessor.
- Imports from `<pkg>` (read-only inspection).

**Forbidden in `audit/<stem>.py`:**

- `skore.evaluate(...)` — duplicates the report under the same key
  and pollutes `summarize()`.
- `project.put(...)` — same.
- Writes outside `scratch/audit/<stem>/` — no `data/` writes, no
  `reports/` writes, no edits to `src/<pkg>/`. The audit is a viewer.
- Mutation of the loaded `report` that survives the cell (e.g.
  monkey-patching skore symbols).

The runner renders every cell's source + last-expression repr +
stdout to the digest. A forbidden call surfaces in the digest (as a
`put` row in a later `summarize()` cell, or as a `**error:**`
section). The contract is *visible*, not invisible.

Sibling read-only consumers (different output shapes, same
discipline): `scratch/<ts>_*.py` probes, `iterate-from-skore`'s
Backlog enrichment walk. See `evaluate-ml-pipeline` § Stop
conditions for the three-consumer rule.

## Stop conditions — read before anything else

- **Read-only against the skore Project.** See § Read-only contract.
  Never `skore.evaluate(...)` or `project.put(...)` in an audit file.
- **`project.get(...)` is by id, not key.** For hub mode, read the
  id from the URL printed by `project.put()`:
  `https://…/<workspace>/<project>/<type-plural>/<N>` → id is
  `skore:report:<type-singular>:<N>` (URL segment is plural; id uses
  the singular — drop the trailing `s`, e.g. `cross-validations` →
  `cross-validation`, `estimators` → `estimator`). Hardcode
  `REPORT_ID` in the audit file — no `summarize()` traversal needed.
  For local mode, read the `"id"` column of `project.summarize()` for
  the matching key row. A `KeyError` from `get("<stem>")` means the
  lookup shape is wrong (get is by id), not that the report is
  missing.
- **Symbol from memory is forbidden.** Any `skore` / `skrub` /
  `sklearn` symbol must come from `python-api` *this turn*. Cache
  hits under `scratch/api/skore/<version>/` count (Shape 0); inline
  memory does not.
- **Agent feature missing → STOP and delegate.** If `ipython` /
  `pyright` aren't importable, do NOT fabricate audit outputs by
  writing `print()` calls as a workaround. Do NOT type
  `pixi add ...` / `uv add ...` yourself — install is owned by
  `python-env-manager` § Agent feature. Request via
  `G-AGENT-FEATURE` (binary: install / skip); resume only when
  python-env-manager returns "ready".
- **Bare expressions, not `print()`.** The runner captures each
  cell's last bare expression via `result.result` and renders its
  `repr`. Wrapping in `print(repr(...))` lands in stdout instead of
  the output section; mixed and harder to scan. Use bare
  expressions; statement-only cells (variable binding) are fine.
- **One audit file per experiment stem (four-way pairing).** No
  `audit_NN_<short_name>_v2.py`. When an experiment is re-run, the
  audit file is **overwritten in place** — same stem, same audit.
- **Executed artifacts go to `scratch/audit/<stem>/`, NOT into
  `audit/`.** Durable artifact is `audit/<stem>.py`; the rendered
  digest is ephemeral.
- **`audit/` is read-only against workspace data.** No writes to
  `data/`, `reports/`, or outside `scratch/audit/<stem>/`.
- **Don't filter warnings in audit cells.** No
  `warnings.filterwarnings(...)` unless the user explicitly asks
  — the runner streams cell stderr into the digest and that's
  signal. See `python-code-style` § Stop conditions.
- **Harness "no clarifying questions" hints do NOT waive
  G-AGENT-FEATURE.** Install gate fires regardless.
- **Post-hoc audit — required before ending the turn.** Walk every
  pre-flight row; surface unfilled Evidence cells.

## Forbidden shortcuts

| Shortcut | Why it's wrong |
|---|---|
| `report = project.get(REPORT_ID); print(repr(report))` | Runner captures bare expressions via `result.result`, not stdout. `print(repr(...))` mixes stdout and output sections. Use `report` on its own line |
| Drop `.frame()` from `report.checks.summarize()` / `report.metrics.summarize()` | `__repr__` of the Display objects is `<…Display at 0x…>`. `.frame()` returns a DataFrame whose repr carries the actual values |
| `project.get(KEY)` raised `KeyError` → re-run `evaluate` + `put` "to refresh" | Lookup shape is wrong (get is by id, not key). Hub: read the id from the URL printed by `put()`. Local: read `summary["id"]` for the matching key row. Never re-run `evaluate` + `put` to recover |
| Write `pixi add --feature agent ipython pyright` directly from this skill | Install commands owned by `python-env-manager`. This skill **requests** via G-AGENT-FEATURE; it does not install |
| Dump the audit `.py` into `scratch/audit/<stem>/` | `.py` is durable in git; `scratch/` is gitignored. Source in `audit/`; digest in `scratch/audit/<stem>/` |
| Register a Jupyter kernel "to be safe" | Current runner is in-process; no kernel. Registering creates an orphan kernelspec |
| Add a fix-up cell that mutates `data/` or `reports/` | Audit files are read-only. State mutations belong in a `scratch/<ts>_*.py` probe or the experiment script |
| Substitute `<SKORE_PROJECT_INIT>` in `audit/<stem>.py` without reading `experiments/<stem>.py` first | Audit must open the same Project. Always Read experiments/<stem>.py this turn and copy the literal Project init block byte-identical (modulo formatting) |
| Hub mode: put `skore.login(mode="hub")` after `skore.Project(...)` | `Project(...)` constructor authenticates at init time; without prior `login`, fails. Order is fixed: login first, Project second |
| § 4 dispatched audit → write scratch probe first to "double-check metrics" | The audit IS the metric-extraction step in § 4. Scratch probes for metrics are the anti-pattern this dispatch replaces |

## Pre-flight — emit before any audit-file write or execution

```
Pre-flight (audit-ml-pipeline):
- [ ] Experiment stem confirmed: <NN_short_name>
      Evidence: journal/NN_<short_name>.md exists AND state ≥ done
                | "n/a — user invoked re-audit on existing stem"
- [ ] Four-way pairing complete:
        journal/NN_<short_name>.md       — design note (state ≥ done)
        experiments/NN_<short_name>.py   — script
        tests/smoke/test_NN_<short_name>.py — smoke test (passing)
        audit/NN_<short_name>.py         — about to be written / refreshed
      Evidence: ls / Glob on each path
- [ ] Report present in skore Project under key=<NN_short_name>
      Evidence: scratch/<ts>_check_report.py probe ran
                project.summarize() this turn; row with
                key == "<NN_short_name>" appears.
                "Run finished, put() landed" is NOT sufficient.
- [ ] Agent feature available:
        `pixi run -e agent ipython -c "print(0)"` exit 0
        `pixi run -e agent pyright --version` exit 0
      Evidence: tool output of each
                | JOURNAL.md Status `agent feature: installed`
                Missing → STOP, delegate to python-env-manager G-AGENT-FEATURE
- [ ] python-api consulted for skore symbols used:
      Project, summarize, get, report.checks.summarize, report.metrics.summarize
      Evidence: Read scratch/api/skore/<version>/<topic>.md (this turn)
                | Write the same (this turn)
                | "n/a — cache hit, file already on disk + Read this turn"
- [ ] Template copy + substitution decided:
        <pkg> → package name from src/<pkg>/
        <NN>_<short_name> → experiment stem
        <SKORE_PROJECT_INIT> → literal block copi