设计与 UI
automl-hyperparameter-optimization
Best practices for AutoML and hyperparameter search with Optuna, Ray Tune, and PyCaret, covering search-space design, validation splits, and leakage prevention. Use when tuning model hyperparameters, setting up a pruned or distributed hyperparameter search, designing a nested validation scheme, or evaluating whether an AutoML leaderboard result is production-ready.
Mindrally · 已有 AI 审核记录
安装与使用
运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。
npx skills add Mindrally/skills --skill automl-hyperparameter-optimization技能说明
name: automl-hyperparameter-optimization description: "Best practices for AutoML and hyperparameter search with Optuna, Ray Tune, and PyCaret, covering search-space design, validation splits, and leakage prevention. Use when tuning model hyperparameters, setting up a pruned or distributed hyperparameter search, designing a nested validation scheme, or evaluating whether an AutoML leaderboard result is production-ready."
AutoML and Hyperparameter Optimization
This skill covers designing sound hyperparameter searches and using AutoML tooling (Optuna, Ray Tune, PyCaret, time-series AutoML libraries) without bypassing problem framing, validation design, or explainability.
Workflow for Running a Hyperparameter Search
- Define the target metric and baseline first — Pick the metric before selecting tooling, and train a simple baseline (linear model, random forest, or naive time-series forecast) with a fixed, minimal search.
- Design the validation scheme — Use nested cross-validation or a final untouched test split for any model-selection claim; use time-aware splits (never shuffled) for time-series problems.
- Fit preprocessing inside the fold — Fit scalers, encoders, and imputers only on the training portion of each fold to prevent leakage.
- Define a structured search space — Use log-scale ranges for learning rates, regularization strength, and tree counts; keep ranges domain-informed rather than arbitrarily broad.
- Choose the right tool — Optuna or Ray Tune for custom training loops with pruning and distributed trials; PyCaret for a quick low-code comparison on a straightforward tabular problem; a time-series-specific library (AutoTS, Merlion, PyAF) when seasonality and horizon handling need first-class support.
- Run with resource limits and pruning — Set a trial or time budget and use early stopping/pruning so bad trials don't consume the full budget.
- Track every run — Log datasets, splits, metric definitions, random seeds, library versions, and the search space itself to MLflow, Weights & Biases, TensorBoard, or an equivalent tracker.
- Report against the baseline — Compare the selected model to the baseline and at least one non-AutoML alternative before calling it production-ready.
Experiment Design
- Define the target metric before choosing tooling — the metric shapes the search space and the pruning strategy, not the other way around.
- Use nested validation (an inner loop for hyperparameter selection, an outer loop for performance estimation) or a final untouched test split whenever reporting a model-selection claim.
- Use time-aware splits for time-series problems — never shuffle across time boundaries, since that leaks future information into training.
- Fit all preprocessing (scalers, encoders, imputers, feature selection) only on the training fold, never on validation or test data.
- Always include simple baselines: a linear/logistic model, a random forest, or — for time series — a naive/seasonal-naive forecast. A complex model that doesn't beat the baseline is not worth the operational cost.
- Use early stopping and resource limits (max trials, wall-clock budget) for expensive searches so a runaway search doesn't consume unbounded compute.
- Prefer structured, domain-informed search spaces over arbitrarily broad grids — a learning rate range of
1e-5to1e-1on a log scale is more useful than0.0001to10on a linear scale.
Search Space Design
- Keep search spaces explicit and reviewed by someone other than the author — an unreviewed space can silently exclude the true optimum or waste budget on implausible regions.
- Use log-scale sampling for learning rates, regularization coefficients, tree counts, and other scale-sensitive hyperparameters.
- Constrain model complexity (max depth, layer width, number of estimators) to keep training time and memory use realistic for the deployment environment.
- Only include preprocessing choices in the search space when they can be applied per-fold without leakage.
- Never tune on the test set — the test set exists solely to report a final, unbiased estimate once tuning is complete.
Tooling
Optuna — custom loops with pruning
Use Optuna for fine-grained control over the training loop, trial pruning, and search algorithms (TPE, CMA-ES).
import optuna
from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.model_selection import cross_val_score, StratifiedKFold
X, y = load_breast_cancer(return_X_y=True)
def objective(trial: optuna.Trial) -> float:
params = {
"n_estimators": trial.suggest_int("n_estimators", 50, 500, log=True),
"max_depth": trial.suggest_int("max_depth", 2, 10),
"learning_rate": trial.suggest_float("learning_rate", 1e-3, 3e-1, log=True),
"subsample": trial.suggest_float("subsample", 0.5, 1.0),
}
model = GradientBoostingClassifier(random_state=42, **params)
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
scores = cross_val_score(model, X, y, cv=cv, scoring="roc_auc")
# Report the running mean for pruning support
trial.report(scores.mean(), step=0)
if trial.should_prune():
raise optuna.TrialPruned()
return scores.mean()
study = optuna.create_study(
direction="maximize",
sampler=optuna.samplers.TPESampler(seed=42),
pruner=optuna.pruners.MedianPruner(n_warmup_steps=5),
)
study.optimize(objective, n_trials=100, timeout=1800)
print("Best AUROC:", study.best_value)
print("Best params:", study.best_params)
- Use
optuna.pruners.MedianPrunerorHyperbandPrunerto stop unpromising trials early, especially for iterative models (gradient boosting, neural networks). - Set both
n_trialsandtimeoutso the search always terminates within budget. - Seed the sampler for reproducibility, and log
study.trials_dataframe()to your experiment tracker.