explore-ml-data
Owns data understanding BEFORE any model is designed. Places and executes `data/eda.py` (a jupytext `# %%` script) via the shared in-process runner, reads the streamed digest, then writes a persisted `data/eda.md` report (plus linked `data/eda_<table>.html` skrub `TableReport` pages) and the `## Data understanding (EDA)` section of `journal/JOURNAL.md`. The point is to surface the dataset facts — shape, dtypes, missingness, cardinality, target balance / skew, datetime / group structure, feature associations — that JUSTIFY the later learner / splitter / metric decisions, so the user understands *why* the modelling choices are made. Uses `skrub.TableReport` for dataframe overviews and the shared runner `audit-ml-pipeline/scripts/run_cells.py`. Stops at "EDA executed, `data/eda.md` + HTML written, JOURNAL EDA section updated." Never designs the model, never edits `src/<pkg>/`, never modifies the user's raw data files. TRIGGER — any of: - `iterate-ml-experiment` § 0 bootstrap, BEFORE the b
probabl-ai · 已有 AI 审核记录
安装与使用
运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。
npx skills add probabl-ai/skills --skill explore-ml-data技能说明
name: explore-ml-data
description: >
Owns data understanding BEFORE any model is designed. Places and
executes data/eda.py (a jupytext # %% script) via the shared
in-process runner, reads the streamed digest, then writes a
persisted data/eda.md report (plus linked data/eda_<table>.html
skrub TableReport pages) and the ## Data understanding (EDA)
section of journal/JOURNAL.md. The point is to surface the
dataset facts — shape, dtypes, missingness, cardinality, target
balance / skew, datetime / group structure, feature associations —
that JUSTIFY the later learner / splitter / metric decisions, so the
user understands why the modelling choices are made. Uses
skrub.TableReport for dataframe overviews and the shared runner
audit-ml-pipeline/scripts/run_cells.py. Stops at "EDA executed,
data/eda.md + HTML written, JOURNAL EDA section updated." Never
designs the model, never edits src/<pkg>/, never modifies the
user's raw data files.
TRIGGER — any of:
iterate-ml-experiment§ 0 bootstrap, BEFORE the baseline design note — the G-EDA gate fires here (run / skip).- The user asks to "explore the data", "do an EDA", "profile the dataset", "what does the data look like", "understand the data".
- A new or changed data source needs (re-)understanding before the next experiment.
SKIP when: the workspace isn't scaffolded / bootstrapped yet —
iterate-ml-experiment § 0 owns bootstrap ordering and will
dispatch here at the G-EDA step; don't run standalone ahead of
scaffolding (route to iterate-ml-experiment / organize-ml- workspace); there is no data to explore yet; the user wants to
inspect a finished run's skore report rather than the raw dataset
(audit-ml-pipeline); the user is past data understanding and wants
pipeline / evaluation mechanics (build-ml-pipeline /
evaluate-ml-pipeline); a pure symbol lookup (python-api); EDA is
already recorded (data/eda.md + the JOURNAL EDA section exist) and
the user is not asking to refresh it.
HOW TO USE: run the Detection step (does data/eda.md + the JOURNAL
EDA section already exist?), emit the Pre-flight checklist as
visible text, read the Stop conditions, then place data/eda.py
from templates/eda.py, execute it via the shared runner, read the
digest, and author data/eda.md + the JOURNAL EDA section. Always
resolve skrub / pandas / polars symbols via python-api, never from
memory.
Explore ML Data
Understand the dataset before designing a model. One project-level
EDA per workspace: an executable data/eda.py, a persisted
data/eda.md narrative, rich data/eda_<table>.html reports, and a
short JOURNAL section that links them. The findings feed the baseline
design note's learner / splitter / metric choices.
Next-step pointers — where you go after this skill
| You came here for… | → next |
|---|---|
| Bootstrap, before the first baseline | → back to iterate-ml-experiment § 0; the EDA findings inform the auto-drafted 01_baseline.md |
| User free-text ("explore the data") | → surface the findings; no further dispatch unless the user asks to model |
| Re-understand a changed data source | → re-run, overwrite data/eda.*, refresh the JOURNAL EDA section |
Always re-emit the Pre-flight checklist with evidence before declaring the turn done.
Where this sits in the loop
EDA is a bootstrap-time gate (G-EDA) owned by this skill and
fired by iterate-ml-experiment § 0 before the baseline design
note. Ordering matters: the dataset facts (class balance, datetime /
group columns, missingness, cardinality) are exactly what justifies
the splitter (G-CV-SPLITTER), the metric default, and the learner
default. Running EDA after the model is designed defeats the purpose.
scaffold → JOURNAL → goal from data/README.md
│
└─► G-EDA (run | skip) ◄── this skill
│ run
└─► data/eda.py → execute → data/eda.md + HTML + JOURNAL §EDA
│
└─► auto-draft 01_baseline.md (cites the EDA findings)
Where things live — visual map
Two locations are kept separate: the raw data source (read-only,
may live anywhere) and the EDA deliverables (always under
<project>/data/).
| Path | Durability | Who writes it | What it holds |
|---|---|---|---|
raw data source (data/, raw/, an absolute path, external) | user-owned, READ-ONLY | the user | The dataset. EDA reads it; never modifies it. May be anywhere — not assumed to be data/ |
data/eda.py | Durable (committed) | This skill, once per workspace | The jupytext # %% EDA cells. Source of truth. Openable as a notebook for the rich view |
data/eda.md | Durable (committed) | This skill (authored from the digest) | The prose narrative: findings + modelling implications that the baseline note cites |
data/eda_<table>.html | Durable (committed) | data/eda.py via TableReport.write_html(...) | The rich, interactive skrub report per table — for the human |
scratch/eda/eda.md | Ephemeral (gitignored), optional | run_cells.py when given a 2nd arg | Per-cell digest the agent reads. Same content as stdout |
journal/JOURNAL.md § Data understanding (EDA) | Durable (committed) | This skill | 2–4 line summary + link to data/eda.md |
Mnemonic: the raw data is read-only and lives wherever the user
keeps it; data/eda.py is source; data/eda.md + the HTML are the
durable deliverables, always under data/; scratch/eda/ and
stdout are the ephemeral run digest.
Read-only-against-raw-data contract
The central rule. Surfaced as the first Stop condition below.
Allowed — this skill writes ONLY (deliverables always under
<project>/data/, created if absent):
data/eda.py— the EDA script (created / overwritten in place).data/eda.md— the authored narrative.data/eda_<table>.html— the skrubTableReportpages.scratch/eda/— the ephemeral digest.- `jou