跳至正文
OpenAgentSkill
浏览已开放 · 目录记录日期 2026-11-21 · 自动更新和账户服务暂时关闭。
金融

ml4t-registry-system

Content-addressed experiment tracking for ML trading models. Use when versioning models, features, or experiment artifacts for reproducibility.

ml4t · 已有 AI 审核记录

安装与使用

运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。

npx skills add ml4t/skills --skill ml4t-registry-system

技能说明


name: ml4t-registry-system description: "Content-addressed experiment tracking for ML trading models. Use when versioning models, features, or experiment artifacts for reproducibility." when_to_use: "Use when running model experiments and need reproducibility, comparison, and audit trail across training runs" dependencies: [] metadata: book_chapters: "11, 12" library: "" paths: ["/schema.py", "/registry.py", "/pipeline.py", "/polars.py", "**/case_study.py"]

Experiment Registry

Without a registry, you overwrite the best model every time you retrain. Content-addressed storage - where hash(config) determines the storage path - makes every experiment reproducible and comparable without manual bookkeeping.

The Problem

A quant runs 50 model configurations. Results go into model_v2_final_FINAL.pkl. Next week, a new run overwrites it. The team cannot answer: which hyperparameters produced the best IC? Was that before or after the feature change? Did we already try alpha=0.01? Without structured tracking, experiments are lost, repeated, and unverifiable.

The Pattern

WRONG

import pickle

# Overwrite on every run - no history, no comparison, no provenance
model.fit(X_train, y_train)
with open("best_model.pkl", "wb") as f:
    pickle.dump(model, f)

# Three weeks later: "Which config was this? What data did it use?"

CORRECT

import hashlib
import json
import sqlite3
from datetime import datetime

def config_hash(config: dict) -> str:
    """Deterministic hash of experiment config."""
    blob = json.dumps(config, sort_keys=True).encode()
    return hashlib.sha256(blob).hexdigest()[:12]

def register_run(db_path: str, config: dict, metrics: dict, predictions_path: str):
    """Register a training run with full provenance."""
    run_hash = config_hash(config)
    conn = sqlite3.connect(db_path)
    conn.execute("""
        CREATE TABLE IF NOT EXISTS training_runs (
            run_hash TEXT PRIMARY KEY,
            config JSON NOT NULL,
            metrics JSON NOT NULL,
            predictions_path TEXT,
            created_at TEXT NOT NULL
        )
    """)
    conn.execute(
        "INSERT OR REPLACE INTO training_runs VALUES (?, ?, ?, ?, ?)",
        (run_hash, json.dumps(config), json.dumps(metrics),
         predictions_path, datetime.now().isoformat()),
    )
    conn.commit()
    return run_hash

# Usage: every config gets a unique, reproducible slot
config = {"model": "ridge", "alpha": 1.0, "features": "momentum_v2"}
run_hash = register_run("registry.db", config, {"ic": 0.04}, f"runs/{config_hash(config)}/predictions.parquet")
# Re-running same config overwrites same slot - idempotent

Registry Schema

Three linked tables capture the full experiment lifecycle:

training_runs          prediction_sets         backtest_runs
+------------+        +----------------+       +--------------+
| run_hash   |<------>| pred_hash      |<----->| bt_hash      |
| config     |   1:N  | run_hash (FK)  |  1:N  | pred_hash(FK)|
| metrics    |        | fold           |       | config       |
| created_at |        | path           |       | metrics      |
+------------+        +----------------+       +--------------+
  • training_runs: one row per unique model config (hash of hyperparams)
  • prediction_sets: one row per fold or time split within a training run
  • backtest_runs: one row per strategy config applied to a prediction set

Content-Addressed Storage

run_log/
  registry.db              # SQLite: all metadata
  models/{config_hash}/    # hash(config) -> directory
    config.json
    metrics.json
    predictions.parquet

The hash is the directory name. Same config always maps to the same directory. No manual naming, no collisions, no "v2_final" suffixes. Query the registry with standard SQL against registry.db.

Guardrails

  • Hash must be deterministic: json.dumps(config, sort_keys=True) - without sort_keys, same config produces different hashes
  • Register per-config as they complete, not in bulk after all finish - a crash at config 49 of 50 loses everything otherwise
  • Never store model weights in the SQLite database - store paths to artifacts on disk
  • Config must capture everything needed to reproduce: model type, hyperparameters, feature version, data version, random seed
  • Old runs are never deleted - mark as superseded, keep for audit trail

Checklist

  • Every experiment has a deterministic config hash
  • Registry stores config, metrics, and artifact paths (not weights in DB)
  • Runs registered incrementally (per-config, not bulk)
  • Same config re-run maps to same hash (idempotent)
  • Top-N query by any metric works against the registry
  • Full provenance: model type, hyperparams, feature version, data version, seed