跳至正文
OpenAgentSkill
浏览已开放 · 目录记录日期 2026-11-21 · 自动更新和账户服务暂时关闭。
开发与测试

ml4t-latent-factors

Extract latent factors from return panels using PCA, IPCA, or autoencoders with proper noise diagnostics. Use when reducing dimensionality or discovering risk structure.

ml4t · 已有静态检查记录

安装与使用

运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。

npx skills add ml4t/skills --skill ml4t-latent-factors

技能说明


name: ml4t-latent-factors description: "Extract latent factors from return panels using PCA, IPCA, or autoencoders with proper noise diagnostics. Use when reducing dimensionality or discovering risk structure." when_to_use: "Use when building factor models from high-dimensional return or characteristic panels" dependencies: [lookahead-bias, walk-forward-cv] paths: ["/feature.py", "/label.py", "/barrier.py", "/store.py", "/horizon.py", "/meta_label.py", "/microstructure.py", "/regime.py", "**/selection.py"] metadata: book_chapters: "14" library: "ml4t-diagnostic"

Latent Factor Extraction

PCA explains 80% of return variance - but variance is not alpha. The first principal component captures market beta, which earns the equity premium, not a tradeable edge. Confusing variance-explained with pricing power is the central mistake.

The Problem

With 400+ published return predictors, hand-picking factors invites overfitting. Latent factor methods (PCA, autoencoders) extract structure directly from data. But three failure modes undermine them:

  1. Variance != pricing - high-variance factors may capture idiosyncratic noise, not compensated risk.
  2. Eigenvector instability - when assets (N) approach time periods (T), sample covariance is dominated by noise. Marchenko-Pastur theory gives the noise boundary.
  3. Full-sample PCA is leakage - fitting PCA on the complete panel, then testing on a held-out period, leaks the covariance structure of the test period into training.

The Pattern

WRONG

from sklearn.decomposition import PCA
import numpy as np

# Fit PCA on FULL return panel, then use factors for prediction
pca = PCA(n_components=5)
factors = pca.fit_transform(returns_panel)  # Full-sample fit = leakage

# "80% variance explained" - but does it predict returns?
print(f"Explained variance: {pca.explained_variance_ratio_.sum():.1%}")
signal = factors[:, 0]  # Assumes first PC predicts returns

CORRECT

from sklearn.decomposition import PCA
import numpy as np

def mp_upper_bound(n_assets, n_periods):
    """Marchenko-Pastur noise threshold for a correlation matrix."""
    gamma = n_assets / n_periods
    return (1 + np.sqrt(gamma)) ** 2

# Walk-forward PCA: fit on training window only
def walk_forward_pca(returns, n_components=5, train_window=504):  # 504 ≈ 2 years
    """Fit per fold; keep only components above that fold's noise bound."""
    factors = np.full((len(returns), n_components), np.nan)
    n_assets = returns.shape[1]
    for t in range(train_window, len(returns)):
        train = returns[t - train_window:t]
        # Standardise on the training fold: the bound is stated for a
        # correlation matrix, so raw return variances are not comparable.
        mu, sd = train.mean(0), train.std(0)
        pca = PCA(n_components=n_components)
        pca.fit((train - mu) / sd)
        scores = pca.transform((returns[t:t + 1] - mu) / sd)[0]
        # Drop this fold's noise components here. Averaging eigenvalues over
        # every fold and filtering once would select using future data.
        keep = pca.explained_variance_ > mp_upper_bound(n_assets, train_window)
        factors[t] = np.where(keep, scores, np.nan)
    return factors

factors = walk_forward_pca(returns)
kept = np.mean(np.any(~np.isnan(factors), axis=0))
print(f"Components above the noise bound in at least one fold: {kept:.0%}")

Method Comparison

MethodStrengthsLimitations
PCAFast, linear, interpretable loadingsStatic betas, variance != pricing
IPCADynamic betas via characteristicsSensitive to characteristic selection
AutoencoderNon-linear factor structureOverfits without adversarial constraints
RP-PCAIncorporates risk-premium signalRequires pricing-error objective

Start with PCA + Marchenko-Pastur filtering. Graduate to IPCA only if characteristics drive time-varying exposures.

Guardrails

  • Walk-forward fit: PCA must be re-fit on each fold's training data - never on the full panel
  • Marchenko-Pastur test: discard components below the random-matrix noise bound
  • Microcap bias: equal-weighted PCA is dominated by small/illiquid stocks - use market-cap weighting or NYSE breakpoints
  • Loading rotation: eigenvectors are not stable across subperiods - don't assign fixed economic labels ("this is momentum")
  • Autoencoder seeds: report results across 10+ random seeds; single-seed results are unreliable

Production Implementation

ml4t-diagnostic provides factor evaluation infrastructure for custom latent factor outputs:

from ml4t.diagnostic.api import cross_sectional_ic_series

# After walk-forward PCA, evaluate factor predictiveness via IC
ic = cross_sectional_ic_series(
    predictions=factor_df,      # DataFrame with date, symbol, prediction
    returns=forward_returns_df,  # DataFrame with date, symbol, forward_return
    date_col="date",
    entity_col="symbol",
)

Checklist

  • PCA fit walk-forward per fold, not on full panel
  • Marchenko-Pastur noise threshold applied to discard noise components
  • Variance-explained distinguished from predictive power (IC tested)
  • Microcap bias controlled (cap-weighted or filtered universe)
  • Results stable across random seeds (for autoencoders)