Gradient boosting on trees, built one residual at a time.
A single tree gets the big picture and misses by a little almost everywhere. That miss, the gap between the data and the prediction, is the residual. Boosting fits a new tree to the residual, folds in a slice of it, and looks at what is left. Then it does the same thing again.
The top panel is the prediction laid over the data. The bottom panel is what is left over, and the dashed line there is the next tree, already fit to those leftovers. Add a tree to fold a fraction of it into the prediction and watch the leftovers shrink.
Prediction over the data
What is left over (residuals)
Test error is climbing. The extra trees are fitting the noise now. This is where you would stop.
Turn the learning rate down and each tree takes a smaller bite, so you need more of them and the curve stays smooth. Turn it up and the prediction lurches, then starts chasing the noise.
Each tree trained on the bottom panel, the leftovers, not on the original targets. A fraction of it, set by the learning rate, got folded into the prediction up top. The leftovers shrank, the next tree fit the smaller leftovers, and the prediction crept toward the data. Stack enough small corrections and a pile of weak trees becomes a sharp model.
F = y.mean() # start flat for _ in range(n_trees): residual = y - F # what's left over tree = DecisionTreeRegressor(max_depth=2).fit(X, residual) F = F + lr * tree.predict(X) # take a small step
The same loop, in five lines. Start flat, fit the leftover, take a small step, repeat.
When you click Add a tree, what is the new tree trained to predict?
You keep adding trees. Training error falls to near zero, but test error bottoms out and starts rising. What is happening?
You have been running gradient descent without the calculus. For squared-error loss, the gradient of the loss with respect to the prediction at each point is:
The negative gradient is the residual. So fitting a tree to the leftovers is fitting a tree to the negative gradient, and folding in a slice of it is one downhill step. The learning rate is the step size. Boosting is gradient descent where each step is a whole tree instead of a single number, the same loop as the Hot or Cold game with larger pieces.
Set the learning rate to 1 and add a few trees. What happens to the prediction, and why does a smaller rate behave better?
Pick the spike. How many trees before the ensemble catches it, and why can no single shallow tree do it alone?
Watch the test error as you add trees. Name the point where you would stop, and say what the trees are doing after it.