sports-modeling-doctrine
Define a sports analysis or prediction question, grain, decision time, baselines, primary metrics, validation, and acceptance criteria before choosing algorithms. Use at the start of any sports modeling project.
WalrusQuant · 已有 AI 审核记录
安装与使用
运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。
npx skills add WalrusQuant/sports-analytic-skills --skill sports-modeling-doctrine技能说明
name: sports-modeling-doctrine description: > Define a sports analysis or prediction question, grain, decision time, baselines, primary metrics, validation, and acceptance criteria before choosing algorithms. Use at the start of any sports modeling project. license: MIT metadata: version: "0.12.0"
Sports Modeling Doctrine
Outcome
Write a modeling charter before acquiring data or fitting models. The charter is a user-owned source of truth for the question, target, timing, evaluation, and shape of done.
Before code, lock the scientific contract:
- what question is being answered;
- what one row represents;
- what is knowable at decision time;
- what baseline counts as evidence of value;
- how success will be measured out of time;
- what result causes acceptance, revision, or stopping.
If these fields are not named, do not fit a model.
When to use this skill
Use it when starting or reframing a sports analysis, deciding what “good” means, reviewing whether a result is evaluable, or preventing exploratory work from quietly becoming a predictive claim.
After the charter, use only the relevant specialist skills: a public-source
loader for acquisition, eda-sports for data understanding, feature-rules for
time-safe predictors, baseline-models for reference models, and the relevant
statistical, predictive, rating, simulation, or validation skill.
Question types
| Type | Core question | Required evidence | Common overreach |
|---|---|---|---|
| Descriptive | What happened? | coverage, denominators, uncertainty | treating the sample as every era/population |
| Explanatory | What is associated? | design, effects, uncertainty, confounding limits | causal language |
| Predictive | What will happen after time T? | time-safe inputs and ordered holdout | explaining coefficients as causes |
| Causal | What changes under an intervention? | identification strategy and assumptions | using prediction alone |
| Ranking | Who is strongest as of T? | future utility, schedule context, rank stability | treating point ranks as certain |
| Simulation | What distribution follows assumptions? | calibrated inputs and sensitivity | reporting simulated precision as observed fact |
Do not mix explanatory and predictive success. A stable association need not improve forecasts, and a useful predictor need not identify a causal mechanism.
Charter workflow
- Write the question in one sentence.
- Name sport, competition, population, era, and exclusions.
- Define grain: game, team-game, player-game, possession, play, pitch, or other.
- Classify the analysis type using the table above.
- For predictive, ranking, or prospective simulation work, define decision
time
Tand forecast horizon precisely. - Define target or estimand, units, label rules, and treatment of ties or missing outcomes.
- Record base rate, null expectation, or incumbent process.
- Name at least one naive and one strong simple baseline.
- Lock the primary metric and its direction before fitting.
- Choose validation that respects event, season, and availability order.
- Define data requirements, minimum coverage, and provenance standards.
- Write acceptance, failure, revision, and stop conditions.
- Record out-of-scope decisions and unsupported uses.
Grain and decision-time contract
For each proposed field, be able to answer:
row grain: <what one row represents>
entity/event key: <natural key>
decision time T: <timestamp or event boundary>
forecast horizon: <what future interval is predicted>
feature available by T: <yes/no and source timestamp>
target observed after T: <yes/no>
paired/dependent rows: <shared games, players, series, seasons>
Sports data routinely contain doubled team-game rows, postgame summaries, revised injury reports, end-of-season aggregates, and ratings updated after the event. Column names do not prove availability. Define an as-of rule.
Baseline ladder
| Level | Example | Purpose |
|---|---|---|
| Null | constant training prevalence or historical mean | proves value beyond no differentiation |
| Structural | venue indicator, prior rank, or league average by context | captures obvious domain structure |
| Strong simple | shifted form, simple rating, or regularized linear/logistic model | tests whether complexity earns its cost |
| Incumbent | current operational forecast or public reference | measures practical replacement value |
Lock baseline definitions, training windows, and missing-data behavior before candidate tuning. All models must be scored on identical held-out rows.
Metric defaults
| Target | Primary candidates | Baseline | Important secondary evidence |
|---|---|---|---|
| Binary outcome | log-loss, Brier | prevalence, venue/simple rating | calibration and fold spread |
| Margin/continuous | MAE, RMSE | historical mean or simple rating | residual distribution and interval coverage |
| Count | Poisson deviance, MAE | historical mean count | dispersion and zero behavior |
| Time to event | proper survival score/concordance | simple survival estimate | calibration by horizon |
| Ranking | future-result correlation or utility | prior rank/rating | rank stability and uncertainty |
| Simulation | distributional score or coverage | simple empirical distribution | sensitivity to assumptions |
Accuracy is rarely sufficient for probability work. Match the metric to the decision, define its direction, and compute it only on genuinely held-out data.
Validation defaults
Ordered sports events usually require expanding-window or rolling-origin
validation. A default season walk-forward design trains on seasons before s
and evaluates on season s. When a sport lacks clean seasons, split by event
time with explicit gaps or embargoes where labels/features overlap.
Random row shuffles are not the default because they can mix future roster, team-strength, schedule, and feature information into training. Group shared contests together,
阅读完整来源 ↗