mlops-observability
Make an ML system a glass box with reproducible runs, MLflow dataset lineage, drift monitoring, alerting, and SHAP explanations. Use when a deployed model needs traceability, monitoring, alerting, or explanation.
Médéric HURIER (Fmind) · 已有静态检查记录
安装与使用
运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。
npx skills add MLOps-Courses/mlops-coding-skills --skill mlops-observability技能说明
name: mlops-observability description: Make an ML system a glass box with reproducible runs, MLflow dataset lineage, drift monitoring, alerting, and SHAP explanations. Use when a deployed model needs traceability, monitoring, alerting, or explanation. license: MIT metadata: author: Médéric HURIER (Fmind) source: github.com/MLOps-Courses/mlops-coding-skills/tree/main/mlops-observability created: 2026-01-25 updated: 2026-10-12
MLOps Observability
Goal
To implement a "Glass Box" system where every result is Reproducible, every asset has Lineage, and system health is Monitored, Alerted on, and Explained.
Prerequisites
- Language: Python 3.14
- Context: Production monitoring and debugging.
- Platform Suggestion: MLflow 3, SHAP, Evidently, ...
Instructions
Preserve existing work and repository visibility. Prepare local changes first; commit, push, change remote settings, publish, or deploy only with user authorization.
1. Guarantee Reproducibility
Consistency is key. For instance:
- Randomness: Set seeds for
random,numpy,torch,tensorflow. - Dependencies:
uv.lockis the reproducibility mechanism for Python. It records the exact resolved version and hash of every direct and transitive dependency, anduv sync --frozeninstalls the subset selected by platform markers and dependency groups without checking lock freshness; useuv lock --checkto check freshness. - Tools:
mise.lockdoes the same job for the binaries that are not Python packages (dprint,gitleaks,trivy,actionlint,zizmor, ...), recording version, URL, and checksum per platform. Commit both lockfiles; between them, "works on my machine" stops being a category of bug. - Builds:
mise run buildis a plainuv buildproducing a wheel and an sdist. Runtime lockfiles do not pin isolated build requirements. Pin or constrain those separately when reproducible distribution builds are required. - Environment: Ship the same locked set into a
dockerimage (uv sync --frozen), so the runtime matches what was tested. - Code: Track the git commit hash for every run, and fail the pipeline on a dirty working tree so a run can always be traced back to a commit.
2. Track Data Lineage
Know the origin of your data. For instance:
- Datasets: Create MLflow Datasets with
mlflow.data.from_pandas. - Logging: Log inputs to MLflow context with
mlflow.log_input. - Store: Keep tracking and registry on a SQL backend (
sqlite:///mlflow.dblocally, Postgres or a tracking server in production). The legacy file store is in maintenance mode and lacks current SQL-backed capabilities. Keep the SQL backend used by the reference package. - Versioning: Version data files (e.g.,
data/v1.csv) or use DVC. - Transformations: Log preprocessing parameters mapping data versions to model versions.
3. Monitoring & Drift Detection
Watch for silent failures. For instance:
- Validation: Gate models against quality thresholds with
mlflow.validate_evaluation_results(MLflow 3). Evaluate the exact positive integer version returned by training before changing an alias. A string version is interpreted as an alias by the reference package, so convert the returned value withint(model_version.version). - Metrics: Convert unsigned counts to floating point at the metric boundary before subtracting predictions from targets. Preserve the domain schemas and test both overpredictions and underpredictions to catch wrapping residuals.
- Drift: Use
evidentlyto comparereference(training) vscurrent(production) data.- Detect feature and prediction distribution shifts. Diagnose concept drift with delayed-label performance and changes in the conditional target relationship; marginal drift alone does not establish it.
- System: Enable MLflow System Metrics (
log_system_metrics=True) for CPU/GPU.
4. Alerting
Don't stare at dashboards. For instance:
- Local: Use
plyerfor desktop notifications during long training runs. - Production: Use
PagerDuty(critical) orSlack(warnings). - Thresholds: Use Static (fixed value) or Dynamic (anomaly detection) rules.
- Action: Alerts must link to a dashboard or playbook.
5. Explainability (XAI)
Trust but verify. For instance:
- Global: Use Feature Importance (e.g., Random Forest) to understand overall logic.
- Local: Use
SHAPvalues to explain individual predictions. - Artifacts: Save explanations (plots/tables) as MLflow artifacts.
6. Infrastructure & Costs
Optimize resources. For instance:
- Tags: Tag runs with
project,env,user. - Costs: Log
run_timeand instance type to estimate ROI.
7. Observability of the Repository Itself
The pipeline that produces the model needs the same treatment.
- One Gate:
mise run all(format -> check -> test -> build) is the signal that a change is releasable. CI runs that exact task, so a green pipeline and a green laptop mean the same thing. - Static Guarantees: The locked versions of Ruff and
tyrun insidemise run check, alongside thepip-audit,gitleaks, andtrivyscans — quality signals you get on every commit, not once a quarter. - Written Down:
AGENTS.mdrecords the commands, the definition of done, and the conventions, so an AI assistant debugging a production incident reads the same runbook a human does.
Self-Correction Checklist
- Seeds: Are random seeds fixed?
- Lockfiles: Are
uv.lockandmise.lockcommitted, and does the image install with--frozen? - Inputs: Are input datasets logged to MLflow, on a SQL-backed store?
- System Metrics: Is
log_system_metricsenabled? - Explanations: Are SHAP values generated and stored as artifacts?
- Alerts: Are thresholds defined for failures?
- Gate: Does
mise run allpass, and does C