explore-ml-data

Quelldokumentation, keine Anweisungen für diese Website. Vor dem Ausführen von Befehlen die Berechtigungen prüfen.

scaffold → JOURNAL → goal from data/README.md
   │
   └─► G-EDA (run | skip)  ◄── this skill
         │ run
         └─► data/eda.py → execute → data/eda.md + HTML + JOURNAL §EDA
   │
   └─► auto-draft 01_baseline.md  (cites the EDA findings)

name: explore-ml-data
description: >
  Owns data understanding BEFORE any model is designed. Places and
  executes `data/eda.py` (a jupytext `# %%` script) via the shared
  in-process runner, reads the streamed digest, then writes a
  persisted `data/eda.md` report (plus linked `data/eda_<table>.html`
  skrub `TableReport` pages) and the `## Data understanding (EDA)`
  section of `journal/JOURNAL.md`. The point is to surface the
  dataset facts — shape, dtypes, missingness, cardinality, target
  balance / skew, datetime / group structure, feature associations —
  that JUSTIFY the later learner / splitter / metric decisions, so the
  user understands *why* the modelling choices are made. Uses
  `skrub.TableReport` for dataframe overviews and the shared runner
  `audit-ml-pipeline/scripts/run_cells.py`. Stops at "EDA executed,
  `data/eda.md` + HTML written, JOURNAL EDA section updated." Never
  designs the model, never edits `src/<pkg>/`, never modifies the
  user's raw data files.

  TRIGGER — any of:
  - `iterate-ml-experiment` § 0 bootstrap, BEFORE the baseline design
    note — the G-EDA gate fires here (run / skip).
  - The user asks to "explore the data", "do an EDA", "profile the
    dataset", "what does the data look like", "understand the data".
  - A new or changed data source needs (re-)understanding before the
    next experiment.

  SKIP when: the workspace isn't scaffolded / bootstrapped yet —
  `iterate-ml-experiment` § 0 owns bootstrap ordering and will
  dispatch here at the G-EDA step; don't run standalone ahead of
  scaffolding (route to `iterate-ml-experiment` / `organize-ml-
  workspace`); there is no data to explore yet; the user wants to
  inspect a finished run's skore report rather than the raw dataset
  (`audit-ml-pipeline`); the user is past data understanding and wants
  pipeline / evaluation mechanics (`build-ml-pipeline` /
  `evaluate-ml-pipeline`); a pure symbol lookup (`python-api`); EDA is
  already recorded (`data/eda.md` + the JOURNAL EDA section exist) and
  the user is not asking to refresh it.

  HOW TO USE: run the Detection step (does `data/eda.md` + the JOURNAL
  EDA section already exist?), emit the Pre-flight checklist as
  visible text, read the Stop conditions, then place `data/eda.py`
  from `templates/eda.py`, execute it via the shared runner, read the
  digest, and author `data/eda.md` + the JOURNAL EDA section. Always
  resolve skrub / pandas / polars symbols via `python-api`, never from
  memory.

---
name: explore-ml-data
description: >
  Owns data understanding BEFORE any model is designed. Places and
  executes `data/eda.py` (a jupytext `# %%` script) via the shared
  in-process runner, reads the streamed digest, then writes a
  persisted `data/eda.md` report (plus linked `data/eda_<table>.html`
  skrub `TableReport` pages) and the `## Data understanding (EDA)`
  section of `journal/JOURNAL.md`. The point is to surface the
  dataset facts — shape, dtypes, missingness, cardinality, target
  balance / skew, datetime / group structure, feature associations —
  that JUSTIFY the later learner / splitter / metric decisions, so the
  user understands *why* the modelling choices are made. Uses
  `skrub.TableReport` for dataframe overviews and the shared runner
  `audit-ml-pipeline/scripts/run_cells.py`. Stops at "EDA executed,
  `data/eda.md` + HTML written, JOURNAL EDA section updated." Never
  designs the model, never edits `src/<pkg>/`, never modifies the
  user's raw data files.

  TRIGGER — any of:
  - `iterate-ml-experiment` § 0 bootstrap, BEFORE the baseline design
    note — the G-EDA gate fires here (run / skip).
  - The user asks to "explore the data", "do an EDA", "profile the
    dataset", "what does the data look like", "understand the data".
  - A new or changed data source needs (re-)understanding before the
    next experiment.

  SKIP when: the workspace isn't scaffolded / bootstrapped yet —
  `iterate-ml-experiment` § 0 owns bootstrap ordering and will
  dispatch here at the G-EDA step; don't run standalone ahead of
  scaffolding (route to `iterate-ml-experiment` / `organize-ml-
  workspace`); there is no data to explore yet; the user wants to
  inspect a finished run's skore report rather than the raw dataset
  (`audit-ml-pipeline`); the user is past data understanding and wants
  pipeline / evaluation mechanics (`build-ml-pipeline` /
  `evaluate-ml-pipeline`); a pure symbol lookup (`python-api`); EDA is
  already recorded (`data/eda.md` + the JOURNAL EDA section exist) and
  the user is not asking to refresh it.

  HOW TO USE: run the Detection step (does `data/eda.md` + the JOURNAL
  EDA section already exist?), emit the Pre-flight checklist as
  visible text, read the Stop conditions, then place `data/eda.py`
  from `templates/eda.py`, execute it via the shared runner, read the
  digest, and author `data/eda.md` + the JOURNAL EDA section. Always
  resolve skrub / pandas / polars symbols via `python-api`, never from
  memory.
---

# Explore ML Data

Understand the dataset before designing a model. One project-level
EDA per workspace: an executable `data/eda.py`, a persisted
`data/eda.md` narrative, rich `data/eda_<table>.html` reports, and a
short JOURNAL section that links them. The findings feed the baseline
design note's learner / splitter / metric choices.

## Next-step pointers — where you go after this skill

| You came here for… | → next |
|---|---|
| Bootstrap, before the first baseline | → back to `iterate-ml-experiment` § 0; the EDA findings inform the auto-drafted `01_baseline.md` |
| User free-text ("explore the data") | → surface the findings; no further dispatch unless the user asks to model |
| Re-understand a changed data source | → re-run, overwrite `data/eda.*`, refresh the JOURNAL EDA section |

Always re-emit the Pre-flight checklist with evidence before
declaring the turn done.

## Where this sits in the loop

EDA is a **bootstrap-time gate (G-EDA)** owned by this skill and
fired by `iterate-ml-experiment` § 0 **before** the baseline design
note. Ordering matters: the dataset facts (class balance, datetime /
group columns, missingness, cardinality) are exactly what justifies
the splitter (`G-CV-SPLITTER`), the metric default, and the learner
default. Running EDA after the model is designed defeats the purpose.

```
scaffold → JOURNAL → goal from data/README.md
   │
   └─► G-EDA (run | skip)  ◄── this skill
         │ run
         └─► data/eda.py → execute → data/eda.md + HTML + JOURNAL §EDA
   │
   └─► auto-draft 01_baseline.md  (cites the EDA findings)
```

## Where things live — visual map

Two locations are kept separate: the **raw data source** (read-only,
may live anywhere) and the **EDA deliverables** (always under
`<project>/data/`).

| Path | Durability | Who writes it | What it holds |
|---|---|---|---|
| raw data source (`data/`, `raw/`, an absolute path, external) | user-owned, **READ-ONLY** | the user | The dataset. EDA reads it; never modifies it. May be anywhere — not assumed to be `data/` |
| `data/eda.py` | **Durable** (committed) | This skill, once per workspace | The jupytext `# %%` EDA cells. Source of truth. Openable as a notebook for the rich view |
| `data/eda.md` | **Durable** (committed) | This skill (authored from the digest) | The prose narrative: findings + **modelling implications** that the baseline note cites |
| `data/eda_<table>.html` | **Durable** (committed) | `data/eda.py` via `TableReport.write_html(...)` | The rich, interactive skrub report per table — for the human |
| `scratch/eda/eda.md` | Ephemeral (gitignored), optional | `run_cells.py` when given a 2nd arg | Per-cell digest the agent reads. Same content as stdout |
| `journal/JOURNAL.md` § Data understanding (EDA) | **Durable** (committed) | This skill | 2–4 line summary + link to `data/eda.md` |

**Mnemonic:** the raw data is *read-only and lives wherever the user
keeps it*; `data/eda.py` is *source*; `data/eda.md` + the HTML are the
*durable deliverables, always under `data/`*; `scratch/eda/` and
stdout are the *ephemeral run digest*.

## Read-only-against-raw-data contract

The central rule. Surfaced as the first Stop condition below.

**Allowed — this skill writes ONLY (deliverables always under
`<project>/data/`, created if absent):**

- `data/eda.py` — the EDA script (created / overwritten in place).
- `data/eda.md` — the authored narrative.
- `data/eda_<table>.html` — the skrub `TableReport` pages.
- `scratch/eda/` — the ephemeral digest.
- `journal/JOURNAL.md` § Data understanding (EDA).

**Forbidden:**

- Modifying, deleting, renaming, re-encoding, or "cleaning" the
  user's raw data files — **wherever they live** (`data/`, another
  folder, an absolute/external path). EDA **reads** them; it never
  rewrites them. Data cleaning is the pipeline's job
  (`build-ml-pipeline`), declared at fit time, not a one-off mutation.
- Writing anywhere outside the five paths above — no `src/<pkg>/`
  edits, no `reports/` writes, no new experiment files.
- Designing the model: no `skore.evaluate(...)`, no `project.put(...)`,
  no learner selection here. EDA *informs* those; it does not make
  them.

## Stop conditions — read before anything else

- **Read-only against the user's raw data.** See § Read-only-
  against-raw-data contract. `data/eda.py` reads the raw files
  (wherever they live) and writes only the `data/eda.*` deliverables.
- **Deliverables always under `<project>/data/`; the raw source is
  separate.** Write `data/eda.py` / `data/eda.md` /
  `data/eda_<table>.html` under `<project>/data/` (create the folder
  if absent). The raw data the script *reads* may live anywhere
  (`data/`, another in-repo folder, an absolute or external path) —
  decouple the two: a `RAW = <LOAD_RAW_DATA>` source vs an `EDA_DIR`
  output. Never assume the raw data is in `data/`.
- **EDA precedes model design (G-EDA).** In bootstrap, the gate fires
  **before** `journal/01_baseline.md` is drafted. It is binary:
  **run** (place + execute `data/eda.py`, write the deliverables) or
  **skip** (record `Status: skipped — <date>` in the JOURNAL section
  and proceed). Do not silently bypass — fire the `AskUserQuestion`.
  Free-text "go fast" / "quick baseline" does NOT resolve it.
- **Agent feature required to execute.** The cell runner needs
  `ipython`. If it is missing and the user chose **run**, STOP and
  delegate to `python-env-manager` § "Agent feature"
  (`G-AGENT-FEATURE`). Do NOT type `pixi add ... ipython` yourself;
  do NOT fabricate EDA output with hand-written `print()`s. If the
  user declines the agent feature, **fall back to the skip path**
  (record `Status: skipped`) — never loop between run and install.
- **Symbol from memory is forbidden.** Any `skrub` / `pandas` /
  `polars` symbol (`TableReport`, `TableReport.json`, `write_html`,
  `column_associations`, the tabular reader, …) must come from
  `python-api` *this turn*. Cache hits under
  `scratch/api/<lib>/<version>/` count; inline memory does not.
  **`TableReport.json()`'s key names are not formally documented and
  drift across skrub versions — confirm them via `python-api` and
  parse defensively (`.get(...)`).**
- **Library-agnostic — read facts off skrub, not pandas/polars.** The
  workspace may use pandas OR polars (G-TABULAR), whose summary
  methods differ (`select_dtypes` doesn't even exist in polars). The
  structured facts come from `skrub` (`TableReport(...).json()`,
  `column_associations`), which accept both. The ONLY library-
  specific line is `RAW = <LOAD_RAW_DATA>`. Do not write
  `df.isna()`/`df.nunique()`/`df.select_dtypes(...)` etc.
- **`skrub.TableReport` for dataframe overviews.** Every table gets a
  `TableReport(RAW, title=..., verbose=0)` written to
  `data/eda_<table>.html` (the user-facing artifact) AND read via
  `.json()` for the digest. `verbose=0` keeps progress prints out of
  the digest.
- **Never end a cell on a bare `TableReport`.** Outside a notebook,
  `repr(TableReport(df))` is the useless `<TableReport: use .open()
  to display>`. Use `report.write_html(...)` (a statement) for the
  HTML, and end cells on **text-friendly** expressions (`RAW.shape`,
  a `dict`/`list` built from `report.json()`,
  `skrub.column_associations(RAW)`) so the digest carries real
  values. Mirrors audit's `.frame()` rule.
- **Never gitignore the whole `data/`; ask about the inputs.** The
  deliverables live in `data/` and must stay committable, so the
  whole `data/` folder must never be in `.gitignore`. If the raw
  inputs should be kept out of git (large / local-only), fire an
  `AskUserQuestion` offering to ignore **specific input patterns**
  (e.g. `data/raw/`, `data/*.parquet`) — default: don't. Then verify
  the deliverables are tracked (`git check-ignore data/eda.md` must
  return nothing). Never auto-edit `.gitignore` — that is
  `organize-ml-workspace`'s to write; surface the patch and ask.
- **One project-level EDA.** A single `data/eda.py` covers the whole
  dataset; multi-table data gets one `TableReport` cell per table
  inside that one file (run the target/structure cells on the
  target-bearing table). No `eda_v2.py`, no per-experiment EDA files,
  not part of the four-way stem pairing. Re-understanding overwrites
  `data/eda.py` in place.
- **Don't design the model here.** No splitter pick, no metric pick,
  no learner pick. Record *implications* in `data/eda.md`; the picks
  happen in their owning gates (`G-CV-SPLITTER`, the baseline note).
- **Harness "no clarifying questions" hints do NOT waive G-EDA or
  G-AGENT-FEATURE.** Both fire regardless.
- **Post-hoc audit — required before ending the turn.** Walk every
  pre-flight row; surface unfilled Evidence cells explicitly.

## Forbidden shortcuts

| Shortcut | Why it's wrong |
|---|---|
| Design the baseline first, EDA "later if there's time" | Inverts G-EDA. The point is to justify the modelling choices *before* making them. EDA runs first in bootstrap |
| End a cell on a bare `TableReport(df)` to "show the report" | Outside a notebook that repr is `<TableReport: use .open() to display>` — zero signal in the digest. Use `write_html(...)` + a text summary built from `report.json()` |
| `print(...)` instead of a bare summary expression | The runner captures bare last-expressions via `result.result`; `print(...)` lands in stdout and is harder to scan. Use bare expressions |
| Use pandas/polars methods (`df.isna()`, `df.nunique()`, `df.select_dtypes(...)`) for the summaries | Breaks on the other library (