The Knobs and the Folds

The Number You Picked Yourself

Back in logistic regression, the model handed you a probability. A 0.72 chance this customer churns. Then you had to decide whether 0.72 counts as a yes. You picked a cutoff. Maybe 0.5. Maybe 0.3, because missing a churner costs more than a false alarm. The model never chose that number. You did.

That cutoff is a hyperparameter. The weights inside the model are parameters. The difference is who sets them.

Parameters are learned from the data while the model trains. The slope of a regression line. The split points in a tree. The coefficients in logistic regression. You never touch them by hand.

Hyperparameters are set before training starts. The classification threshold. The \(k\) in k-nearest neighbours. A tree's maximum depth. The regularization strength. The number of trees in a forest. One is found by the algorithm. The other is found by you, or by a search you run on your behalf.

Grid Search: Brute Force With a Tidy Name

You rarely know the right hyperparameters up front. So you guess a few candidates for each and try them all. Three values of regularization strength, four values of tree depth. Twelve combinations. Train a model for each, keep the one that scores best. That is grid search.

The cost is the product of the candidate counts, not the sum. Add one more knob and the grid multiplies:

Five hyperparameters with five values each is 3125 models. This is why people reach for randomized search, which samples combinations at random instead of trying every one. It usually finds something nearly as good for a fraction of the work.

Where the Trap Hides

To pick the best combination, you score each one. Score it on what? Not the test set. Spend the test set here and it is no longer unseen, and the final number you report becomes a lie. So you hold out a separate validation set and score on that.

But you try many combinations. Twenty, a hundred, three thousand. Score that many models on one fixed validation set and one of them looks great by luck. You did not find the best model. You found the model that happened to fit the noise in that particular split. You overfit to the validation set. The more you search, the worse it gets.

The fix is to stop trusting a single split.

Cross-Validation: Rotate the Split

Instead of one validation set, rotate. Split the training data into \(K\) parts, called folds. Train on \(K-1\) of them, score on the one left out. Do it \(K\) times, each fold taking its turn as the held-out part. Average the \(K\) scores. Now every point has been used for both training and scoring, and your estimate rests on \(K\) measurements instead of one. That is K-fold cross-validation.

K-Fold

from sklearn.model_selection import KFold, cross_val_score
from sklearn.tree import DecisionTreeClassifier

model = DecisionTreeClassifier(max_depth=5)
kf = KFold(n_splits=5, shuffle=True, random_state=42)

scores = cross_val_score(model, X, y, cv=kf)
print(scores.mean(), scores.std())

Push \(K\) to its limit and each fold is a single row. Train on everything except one point, predict that point, repeat \(n\) times. This is leave-one-out. It wrings the most out of a small dataset and gives an almost unbiased estimate. It also trains \(n\) models. On anything large it is too slow to bother with.

Leave-One-Out

from sklearn.model_selection import LeaveOneOut, cross_val_score

loo = LeaveOneOut()
scores = cross_val_score(model, X, y, cv=loo)   # trains len(X) models
print(scores.mean())

Plain K-fold splits at random. If your classes are imbalanced, say 8 percent churn, a random fold can land with almost no churners in it, and that fold's score means nothing. Stratified K-fold keeps the class proportions the same in every fold. For classification, especially imbalanced classification, this is the default you want.

Stratified K-Fold

from sklearn.model_selection import StratifiedKFold, cross_val_score

skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
scores = cross_val_score(model, X, y, cv=skf)   # class ratio kept per fold
print(scores.mean(), scores.std())

Grid search and cross-validation are usually one move. You hand the search a cross-validator and it scores every combination by averaging across folds, not on one fragile split. That is the whole point of GridSearchCV.

Grid Search + Cross-Validation

from sklearn.model_selection import GridSearchCV, StratifiedKFold

grid = {'max_depth': [3, 5, 7], 'min_samples_leaf': [1, 5, 10]}
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

search = GridSearchCV(DecisionTreeClassifier(), grid, cv=cv)
search.fit(X, y)
print(search.best_params_, search.best_score_)