跳至正文
OpenAgentSkill
浏览已开放 · 目录记录日期 2026-11-21 · 自动更新和账户服务暂时关闭。
其他技能

mlops-observability

Make an ML system a glass box with reproducible runs, MLflow dataset lineage, drift monitoring, alerting, and SHAP explanations. Use when a deployed model needs traceability, monitoring, alerting, or explanation.

Médéric HURIER (Fmind) · 已有静态检查记录

安装与使用

运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。

npx skills add MLOps-Courses/mlops-coding-skills --skill mlops-observability

技能说明


name: mlops-observability description: Make an ML system a glass box with reproducible runs, MLflow dataset lineage, drift monitoring, alerting, and SHAP explanations. Use when a deployed model needs traceability, monitoring, alerting, or explanation. license: MIT metadata: author: Médéric HURIER (Fmind) source: github.com/MLOps-Courses/mlops-coding-skills/tree/main/mlops-observability created: 2026-01-25 updated: 2026-10-12

MLOps Observability

Goal

To implement a "Glass Box" system where every result is Reproducible, every asset has Lineage, and system health is Monitored, Alerted on, and Explained.

Prerequisites

  • Language: Python 3.14
  • Context: Production monitoring and debugging.
  • Platform Suggestion: MLflow 3, SHAP, Evidently, ...

Instructions

Preserve existing work and repository visibility. Prepare local changes first; commit, push, change remote settings, publish, or deploy only with user authorization.

1. Guarantee Reproducibility

Consistency is key. For instance:

  1. Randomness: Set seeds for random, numpy, torch, tensorflow.
  2. Dependencies: uv.lock is the reproducibility mechanism for Python. It records the exact resolved version and hash of every direct and transitive dependency, and uv sync --frozen installs the subset selected by platform markers and dependency groups without checking lock freshness; use uv lock --check to check freshness.
  3. Tools: mise.lock does the same job for the binaries that are not Python packages (dprint, gitleaks, trivy, actionlint, zizmor, ...), recording version, URL, and checksum per platform. Commit both lockfiles; between them, "works on my machine" stops being a category of bug.
  4. Builds: mise run build is a plain uv build producing a wheel and an sdist. Runtime lockfiles do not pin isolated build requirements. Pin or constrain those separately when reproducible distribution builds are required.
  5. Environment: Ship the same locked set into a docker image (uv sync --frozen), so the runtime matches what was tested.
  6. Code: Track the git commit hash for every run, and fail the pipeline on a dirty working tree so a run can always be traced back to a commit.

2. Track Data Lineage

Know the origin of your data. For instance:

  1. Datasets: Create MLflow Datasets with mlflow.data.from_pandas.
  2. Logging: Log inputs to MLflow context with mlflow.log_input.
  3. Store: Keep tracking and registry on a SQL backend (sqlite:///mlflow.db locally, Postgres or a tracking server in production). The legacy file store is in maintenance mode and lacks current SQL-backed capabilities. Keep the SQL backend used by the reference package.
  4. Versioning: Version data files (e.g., data/v1.csv) or use DVC.
  5. Transformations: Log preprocessing parameters mapping data versions to model versions.

3. Monitoring & Drift Detection

Watch for silent failures. For instance:

  1. Validation: Gate models against quality thresholds with mlflow.validate_evaluation_results (MLflow 3). Evaluate the exact positive integer version returned by training before changing an alias. A string version is interpreted as an alias by the reference package, so convert the returned value with int(model_version.version).
  2. Metrics: Convert unsigned counts to floating point at the metric boundary before subtracting predictions from targets. Preserve the domain schemas and test both overpredictions and underpredictions to catch wrapping residuals.
  3. Drift: Use evidently to compare reference (training) vs current (production) data.
    • Detect feature and prediction distribution shifts. Diagnose concept drift with delayed-label performance and changes in the conditional target relationship; marginal drift alone does not establish it.
  4. System: Enable MLflow System Metrics (log_system_metrics=True) for CPU/GPU.

4. Alerting

Don't stare at dashboards. For instance:

  1. Local: Use plyer for desktop notifications during long training runs.
  2. Production: Use PagerDuty (critical) or Slack (warnings).
  3. Thresholds: Use Static (fixed value) or Dynamic (anomaly detection) rules.
  4. Action: Alerts must link to a dashboard or playbook.

5. Explainability (XAI)

Trust but verify. For instance:

  1. Global: Use Feature Importance (e.g., Random Forest) to understand overall logic.
  2. Local: Use SHAP values to explain individual predictions.
  3. Artifacts: Save explanations (plots/tables) as MLflow artifacts.

6. Infrastructure & Costs

Optimize resources. For instance:

  1. Tags: Tag runs with project, env, user.
  2. Costs: Log run_time and instance type to estimate ROI.

7. Observability of the Repository Itself

The pipeline that produces the model needs the same treatment.

  1. One Gate: mise run all (format -> check -> test -> build) is the signal that a change is releasable. CI runs that exact task, so a green pipeline and a green laptop mean the same thing.
  2. Static Guarantees: The locked versions of Ruff and ty run inside mise run check, alongside the pip-audit, gitleaks, and trivy scans — quality signals you get on every commit, not once a quarter.
  3. Written Down: AGENTS.md records the commands, the definition of done, and the conventions, so an AI assistant debugging a production incident reads the same runbook a human does.

Self-Correction Checklist

  • Seeds: Are random seeds fixed?
  • Lockfiles: Are uv.lock and mise.lock committed, and does the image install with --frozen?
  • Inputs: Are input datasets logged to MLflow, on a SQL-backed store?
  • System Metrics: Is log_system_metrics enabled?
  • Explanations: Are SHAP values generated and stored as artifacts?
  • Alerts: Are thresholds defined for failures?
  • Gate: Does mise run all pass, and does C
阅读完整来源 ↗