跳至正文
OpenAgentSkill
浏览已开放 · 目录记录日期 2026-11-21 · 自动更新和账户服务暂时关闭。
开发与测试

mlops-industrialization

Convert notebook prototypes into a distributable Python package with a src layout, a domain/io/application split, and validated OmegaConf plus Pydantic configuration. Use when moving code out of notebooks or designing entrypoints.

Médéric HURIER (Fmind) · 已有静态检查记录

安装与使用

运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。

npx skills add MLOps-Courses/mlops-coding-skills --skill mlops-industrialization

技能说明


name: mlops-industrialization description: Convert notebook prototypes into a distributable Python package with a src layout, a domain/io/application split, and validated OmegaConf plus Pydantic configuration. Use when moving code out of notebooks or designing entrypoints. license: MIT metadata: author: Médéric HURIER (Fmind) source: github.com/MLOps-Courses/mlops-coding-skills/tree/main/mlops-industrialization created: 2026-01-25 updated: 2026-10-12

MLOps Industrialization

Goal

To convert experimental code (notebooks/scripts) into a high-quality, distributable Python package. This skill enforces the src/ layout, a Hybrid Paradigm (OOP structure + Functional purity), and Strict Configuration to ensure scalability, security, and maintainability.

Prerequisites

  • Language: Python 3.14
  • Manager: uv
  • Context: Moving from notebooks/ to src/.

Instructions

Preserve existing work and repository visibility. Prepare local changes first; commit, push, change remote settings, publish, or deploy only with user authorization.

1. Packaging Structure (src Layout)

Adopt the src layout to prevent import errors and separate source from tooling.

  1. Directory Tree:

    my-project/
    ├── pyproject.toml       # Dependencies & Metadata
    ├── uv.lock              # Pinned Python dependencies
    ├── mise.toml            # Task vocabulary & pinned tools
    ├── mise.lock            # Pinned tool binaries
    ├── AGENTS.md            # Instructions for AI agents
    ├── README.md
    └── src/
        └── my_package/      # Main package directory
            ├── __init__.py
            ├── io/          # Side-effects (Datasets, APIs)
            ├── domain/      # Pure business logic (Models, Features)
            └── application/ # Orchestration (Training loops, Inference)
    
  2. Configuration: Use pyproject.toml for all build metadata and dependencies.

2. Modularity & Paradigm (Hybrid Style)

Balance structure with predictability.

  1. Domain Layer (Pure):
    • Rule: Code here must be deterministic and free of side effects (no I/O).
    • Use Case: Feature transformations, Model architecture definitions.
    • Style: Functional (pure functions) or Immutable Objects (dataclasses).
  2. I/O Layer (Impure):
    • Rule: Isolate external interactions here.
    • Use Case: Loading data from S3, saving models to disk, logging to MLflow.
    • Style: OOP (Classes to manage connections/state).
  3. Application Layer (Orchestration):
    • Rule: Wire Domain and I/O together.
    • Use Case: Tuning, Training, Inference, Evaluation, etc.

3. Application Entrypoints

Create standard, installable CLI tools.

  1. Define Script: Create src/my_package/scripts.py with a main() function.

  2. Register: Add to pyproject.toml:

    [project.scripts]
    my-tool = "my_package.scripts:main"
    
  3. CLI Execution:

    • Dev: uv run my-tool (No install needed).
    • Prod: pip install . -> my-tool (Installed on PATH).
  4. Guard: Always use if __name__ == "__main__": in scripts to prevent execution on import.

4. Configuration Management

Decouple settings from code using OmegaConf (Parsing) and Pydantic (Validation).

  1. Define Schema (Pydantic):

    • Create a class that defines expected types and defaults.
    from pydantic import BaseModel
    
    
    class TrainingConfig(BaseModel):
        batch_size: int = 32
        learning_rate: float = 0.001
        use_gpu: bool = False
    
  2. Parse & Validate (OmegaConf): Load YAML, merge dotlist CLI values if your application supports them, then resolve and validate.

    import omegaconf
    
    conf = omegaconf.OmegaConf.load("config.yaml")
    cli_conf = omegaconf.OmegaConf.from_cli()
    merged = omegaconf.OmegaConf.merge(conf, cli_conf)
    cfg = TrainingConfig.model_validate(
        omegaconf.OmegaConf.to_container(merged, resolve=True, throw_on_missing=True)
    )
    
  3. Secrets: Use environment variables or pydantic-settings; never commit credentials.

5. MLflow Services as I/O Objects

Tracking is a side effect, so it belongs in the I/O layer behind a small, configurable service.

  1. Backend: Default the tracking and registry URIs to a SQL store — sqlite:///mlflow.db locally, a Postgres or HTTP tracking server in production. The legacy file store is in maintenance mode and disabled by default in the supported MLflow version. Use the SQL-backed store from the reference package.
  2. Configuration, not constants: Expose tracking_uri, registry_uri, experiment_name, and autolog as validated fields, so the same package runs against a laptop database and a shared server without a code change.
  3. Lifecycle: Give the service explicit start()/stop() methods called by the application layer, never at import time.

6. Documentation & Quality

Make code usable and maintainable.

  1. Docstrings: Use Google Style docstrings for all modules, classes, and functions.
import numpy as np
from sklearn.metrics import accuracy_score


def calculate_metric(y_true: np.ndarray, y_pred: np.ndarray) -> float:
    """Calculates the accuracy score.

    Args:
        y_true: Ground truth labels.
        y_pred: Predicted labels.

    Returns:
        The accuracy as a float between 0 and 1.
    """
    return float(accuracy_score(y_true, y_pred))
  1. Type Hints: Use modern Python typing (list[str], X | Y) everywhere; ty (0.0.69+) checks them.
  2. Instructions: Record the layer boundaries, the naming conventions, and the exact commands in AGENTS.md so assistants stop guessing where new code belongs.
  3. Gate: mise run all (format -> check -> test -> build) must pass before a refactor is considered finished — see mlops-validation.

7. Best Practices Summary

  • Config != Code: Never hard
阅读完整来源 ↗