You need an apartment in Würzburg. One strategy is to collect every listing, build a complete picture of the market, calculate the optimal sequence of viewings, and then act. Another is to open the site, send three messages today, follow up tomorrow, and pick the best response you get.
The first strategy is more correct. It is also slower, and the market does not wait. The apartment your calculation identified as optimal was rented while you were still optimizing. The person using the second strategy signed a lease on Tuesday.
This is the difference in spirit between two versions of gradient descent. One computes carefully, using all available data before each step. The other acts on a small random sample, moves fast, and accepts that each step will be imprecise. In machine learning, the second version almost always wins.
The Terrain
Training a model is a search problem. You are looking for the values of its parameters that produce the lowest possible prediction error. Visualize that error as a landscape. The horizontal axes are parameter values. The height at any point is the loss: how wrong the model currently is. Your goal is to reach the lowest point in that landscape.
Gradient descent navigates this terrain by feel. At each step, it checks the slope at the current position, which direction goes downhill fastest?, and takes one step that way. The update rule:
α is the learning rate: the size of each step. The gradient ∂L/∂w gives the local slope. Move against it and you move downhill. Repeat until the loss stops decreasing. If you have seen the linear regression module, you have already seen this loop.
The Nearest Valley Is Not the Deepest One
The loss landscape of a real model is not a single smooth bowl. It is closer to a mountain range: multiple valleys, ridges, and plateaus where the slope is nearly zero. A ball rolled carefully downhill will find the nearest valley and stop. It has no reason to leave.
This is a local minimum. A point where every direction looks uphill, but a better solution exists somewhere else on the landscape. Batch gradient descent, which computes an exact gradient on all training data before each step, is vulnerable to this. It follows the slope precisely and gets stuck.
On simple problems like linear regression the loss surface is a single smooth bowl with one minimum and this does not matter. On the kind of landscapes that arise when training neural networks, with millions of parameters and complex interactions, local minima are everywhere and precision alone does not save you.
The Sloppy Version
Stochastic gradient descent changes one thing: instead of computing the gradient on all the data, it picks a random sample and computes the gradient on that. The result is a noisy estimate of the true gradient. The step you take is not exactly downhill. It is roughly downhill, with some randomness in it.
The practical version used in most real systems is mini-batch SGD: a small random subset of the data, typically 32 to 512 examples. The dataset is shuffled before each pass through it, so the sequence of mini-batches is different every time. The gradient estimates are imprecise. The path through the landscape is jittery.
That jitter is the point. A ball that wobbles occasionally gets nudged out of shallow valleys. It overshoots small local minima. It keeps moving rather than settling. On a landscape with many imperfect valleys, a slightly noisy path finds better solutions than a perfectly smooth one.
There is a second benefit. Because each step is computed on a different random sample, the model is constantly exposed to different parts of the data. It cannot memorize one particular pattern and call it done. The noise acts as a form of regularization, the model becomes harder to overfit.
This is the part that surprises most people. You would expect precision to lead to better results. In optimization over complex landscapes, it often does not. The sloppiness is not a compromise forced by limited computing power. It is a design choice, kept deliberately after people noticed it works better.
See It Get Stuck
Here is a loss landscape with two valleys. The left one is shallow. The deep one on the right is the global minimum, the answer you actually want. Drop the ball. Batch gradient descent rolls into the nearest valley and stops, even though a better one sits next door. Switch to the stochastic version and drop again. The noise shakes the ball until it stumbles over the ridge and into the deeper valley.
Every Major AI System Was Trained This Way
Language models. Image generators. Recommendation engines. Fraud detection systems. Every one of these was trained with some version of stochastic gradient descent. Not because the teams could not afford to compute the exact gradient, but because they knew the noisy version produces better results.
The step from "minimize error on training data" to "build something that works on new data" is not automatic. Precision on the training set is not the same as usefulness in the world. The noise in SGD is one of the main mechanisms that bridges that gap.
Batch gradient descent finds the deepest valley in the training landscape. Stochastic gradient descent finds a valley that holds up across many slightly different samples of the world. The second is almost always what you want.
For the curious: momentum, adaptive rates, and why the landscape is stranger than you think
Momentum
Plain SGD moves based only on the current gradient. Momentum adds memory: the update step carries some velocity from the previous step, like a ball that keeps rolling in roughly the same direction even when the slope changes slightly. This smooths the path and helps cross small uphill bumps without getting stopped by them.
The Adam Optimizer
Adam combines momentum with adaptive learning rates: each parameter gets its own step size, adjusted based on how much that parameter has been changing. Parameters updated frequently get smaller steps. Parameters that have barely moved get larger ones. Adam is the default optimizer for most modern deep learning. When a paper says "we trained with Adam," this is what it means.
Learning Rate Schedules
A fixed learning rate is always a compromise. Start too large and you overshoot. End too large and you never settle. Schedules begin with a larger rate to explore the landscape quickly, then decay it over time so the model can converge precisely toward the end of training. Warmup, cosine decay, step decay: different schedules for different problems, the same logic underneath all of them.
The Landscape Is Stranger Than You Think
In very high-dimensional spaces, local minima are less common than intuition suggests. More common are saddle points: locations where the surface curves downward in some dimensions and upward in others. Gradient descent can stall at saddle points because the gradient is nearly zero there. SGD's noise tends to escape them naturally, which is another reason the noisy version outperforms the exact one on large models.
A model you use every day got its parameters from this process.
Pick any AI product you interact with regularly. Behind its outputs is a loss landscape, a learning rate someone chose, and millions of noisy gradient steps. If the model surprises you, or fails you, the landscape is often where the explanation lives.