Skip to content
OpenAgentSkill
Browsing is open · Catalog captured 2026-11-21 · Automatic updates and account services are paused.
Data & analytics

ml4t-validate-data

Systematic data quality validation before modeling. Use when checking for gaps, outliers, stale prices, or schema violations in datasets.

ml4t · Static checks on file

Installation

Read the source and dependencies before running a command. Copying does not install or execute anything.

npx skills add ml4t/skills --skill ml4t-validate-data

About this skill


name: ml4t-validate-data description: "Systematic data quality validation before modeling. Use when checking for gaps, outliers, stale prices, or schema violations in datasets." when_to_use: "Use when ingesting new data or before any model training run" dependencies: [fetch-data] metadata: book_chapters: "2, 3" library: "ml4t-data" paths: ["/data.py", "/fetch.py", "/bars.py", "/universe.py", "/calendar.py", "/futures.py", "/export.py", "/synthetic.py"]

Validate Data

Unvalidated data silently corrupts models - a single unadjusted stock split can make a momentum signal look 10x stronger than reality.

The Problem

Financial data arrives with missing values, duplicate timestamps, unadjusted corporate actions, stale prices, and impossible OHLC relationships. Using raw data without checks means your model trains on artifacts. A 50% overnight return that is actually a 2:1 split will dominate any feature that touches price changes. You will not see this in your loss function - the model happily fits the noise.

The Pattern

WRONG

import polars as pl

# Trust the data, start modeling immediately
df = pl.read_parquet("prices.parquet")
features = df.with_columns(ret=pl.col("close").pct_change())
model.fit(features)  # Trained on splits, gaps, duplicates

CORRECT

import polars as pl

def validate_ohlcv(df: pl.DataFrame) -> dict[str, int]:
    """Run standard OHLCV quality checks. Returns issue counts."""
    issues = {}

    # OHLC consistency: high >= low, close within [low, high]
    issues["high_lt_low"] = df.filter(pl.col("high") < pl.col("low")).height
    issues["close_outside_hl"] = df.filter(
        (pl.col("close") > pl.col("high")) | (pl.col("close") < pl.col("low"))
    ).height

    # Missing data
    issues["null_close"] = df["close"].null_count()
    issues["zero_volume"] = df.filter(pl.col("volume") == 0).height

    # Extreme returns (likely unadjusted splits)
    rets = df.sort("symbol", "timestamp").with_columns(
        pl.col("close").pct_change().over("symbol").alias("ret")
    )
    issues["extreme_returns"] = rets.filter(pl.col("ret").abs() > 0.5).height

    # Duplicate timestamps
    issues["duplicates"] = (
        df.group_by("symbol", "timestamp").len()
        .filter(pl.col("len") > 1).height
    )

    # Stale prices (5+ identical closes in a row)
    stale = df.sort("symbol", "timestamp").with_columns(
        (pl.col("close") == pl.col("close").shift(1)).over("symbol").alias("same")
    )
    issues["stale_5d"] = stale.filter(
        pl.col("same").rolling_sum(5).over("symbol") >= 5
    ).height

    # Report
    for check, count in issues.items():
        if count > 0:
            print(f"  FAIL: {check} = {count}")
    return issues

df = pl.read_parquet("prices.parquet")
issues = validate_ohlcv(df)
assert issues["high_lt_low"] == 0, "OHLC consistency violated"
assert issues["duplicates"] == 0, "Duplicate timestamps found"

Corporate Action Detection

Flag likely unadjusted splits: overnight return >40% with no corresponding volume spike. Filter for abs(ret) > 0.4 AND volume / rolling_mean(volume, 20) < 3.0 - real moves come with volume, splits do not.

Guardrails

  • Run validation before every model training, not just at initial load
  • Extreme returns (>50%) are almost always data errors, not real moves
  • Zero-volume days may be valid (halts) or errors - check per-exchange rules
  • Stale prices for 5+ consecutive days indicate a dead feed, not a flat market
  • Timestamps must be timezone-aware - naive datetimes cause silent alignment errors

Production Implementation

ml4t-data validates provider output during fetches and storage-backed ingest:

from ml4t.data import DataManager

dm = DataManager()
df = dm.fetch("SPY", start="2015-01-01", end="2024-12-31", provider="yahoo")
# Returned OHLCV is validated before it reaches your modeling code

Checklist

  • OHLC consistency verified (high >= low, close within range)
  • Missing values counted and below threshold
  • Duplicate timestamps removed
  • Extreme returns flagged and cross-referenced with corporate actions
  • Stale prices detected
  • Timestamps are timezone-aware