mlops-industrialization
Convert notebook prototypes into a distributable Python package with a src layout, a domain/io/application split, and validated OmegaConf plus Pydantic configuration. Use when moving code out of notebooks or designing entrypoints.
Médéric HURIER (Fmind) · 已有静态检查记录
安装与使用
运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。
npx skills add MLOps-Courses/mlops-coding-skills --skill mlops-industrialization技能说明
name: mlops-industrialization description: Convert notebook prototypes into a distributable Python package with a src layout, a domain/io/application split, and validated OmegaConf plus Pydantic configuration. Use when moving code out of notebooks or designing entrypoints. license: MIT metadata: author: Médéric HURIER (Fmind) source: github.com/MLOps-Courses/mlops-coding-skills/tree/main/mlops-industrialization created: 2026-01-25 updated: 2026-10-12
MLOps Industrialization
Goal
To convert experimental code (notebooks/scripts) into a high-quality, distributable Python package. This skill enforces the src/ layout, a Hybrid Paradigm (OOP structure + Functional purity), and Strict Configuration to ensure scalability, security, and maintainability.
Prerequisites
- Language: Python 3.14
- Manager:
uv - Context: Moving from
notebooks/tosrc/.
Instructions
Preserve existing work and repository visibility. Prepare local changes first; commit, push, change remote settings, publish, or deploy only with user authorization.
1. Packaging Structure (src Layout)
Adopt the src layout to prevent import errors and separate source from tooling.
-
Directory Tree:
my-project/ ├── pyproject.toml # Dependencies & Metadata ├── uv.lock # Pinned Python dependencies ├── mise.toml # Task vocabulary & pinned tools ├── mise.lock # Pinned tool binaries ├── AGENTS.md # Instructions for AI agents ├── README.md └── src/ └── my_package/ # Main package directory ├── __init__.py ├── io/ # Side-effects (Datasets, APIs) ├── domain/ # Pure business logic (Models, Features) └── application/ # Orchestration (Training loops, Inference) -
Configuration: Use
pyproject.tomlfor all build metadata and dependencies.
2. Modularity & Paradigm (Hybrid Style)
Balance structure with predictability.
- Domain Layer (Pure):
- Rule: Code here must be deterministic and free of side effects (no I/O).
- Use Case: Feature transformations, Model architecture definitions.
- Style: Functional (pure functions) or Immutable Objects (dataclasses).
- I/O Layer (Impure):
- Rule: Isolate external interactions here.
- Use Case: Loading data from S3, saving models to disk, logging to MLflow.
- Style: OOP (Classes to manage connections/state).
- Application Layer (Orchestration):
- Rule: Wire Domain and I/O together.
- Use Case: Tuning, Training, Inference, Evaluation, etc.
3. Application Entrypoints
Create standard, installable CLI tools.
-
Define Script: Create
src/my_package/scripts.pywith amain()function. -
Register: Add to
pyproject.toml:[project.scripts] my-tool = "my_package.scripts:main" -
CLI Execution:
- Dev:
uv run my-tool(No install needed). - Prod:
pip install .->my-tool(Installed on PATH).
- Dev:
-
Guard: Always use
if __name__ == "__main__":in scripts to prevent execution on import.
4. Configuration Management
Decouple settings from code using OmegaConf (Parsing) and Pydantic (Validation).
-
Define Schema (Pydantic):
- Create a class that defines expected types and defaults.
from pydantic import BaseModel class TrainingConfig(BaseModel): batch_size: int = 32 learning_rate: float = 0.001 use_gpu: bool = False -
Parse & Validate (OmegaConf): Load YAML, merge dotlist CLI values if your application supports them, then resolve and validate.
import omegaconf conf = omegaconf.OmegaConf.load("config.yaml") cli_conf = omegaconf.OmegaConf.from_cli() merged = omegaconf.OmegaConf.merge(conf, cli_conf) cfg = TrainingConfig.model_validate( omegaconf.OmegaConf.to_container(merged, resolve=True, throw_on_missing=True) ) -
Secrets: Use environment variables or
pydantic-settings; never commit credentials.
5. MLflow Services as I/O Objects
Tracking is a side effect, so it belongs in the I/O layer behind a small, configurable service.
- Backend: Default the tracking and registry URIs to a SQL store —
sqlite:///mlflow.dblocally, a Postgres or HTTP tracking server in production. The legacy file store is in maintenance mode and disabled by default in the supported MLflow version. Use the SQL-backed store from the reference package. - Configuration, not constants: Expose
tracking_uri,registry_uri,experiment_name, andautologas validated fields, so the same package runs against a laptop database and a shared server without a code change. - Lifecycle: Give the service explicit
start()/stop()methods called by the application layer, never at import time.
6. Documentation & Quality
Make code usable and maintainable.
- Docstrings: Use Google Style docstrings for all modules, classes, and functions.
import numpy as np
from sklearn.metrics import accuracy_score
def calculate_metric(y_true: np.ndarray, y_pred: np.ndarray) -> float:
"""Calculates the accuracy score.
Args:
y_true: Ground truth labels.
y_pred: Predicted labels.
Returns:
The accuracy as a float between 0 and 1.
"""
return float(accuracy_score(y_true, y_pred))
- Type Hints: Use modern Python typing (
list[str],X | Y) everywhere;ty(0.0.69+) checks them. - Instructions: Record the layer boundaries, the naming conventions, and the exact commands in
AGENTS.mdso assistants stop guessing where new code belongs. - Gate:
mise run all(format -> check -> test -> build) must pass before a refactor is considered finished — see mlops-validation.
7. Best Practices Summary
- Config != Code: Never hard