Class 5, module two of three. One dial, five weights, and a bar that never quite reaches zero.
Two advertising channels. Google Search and Google Shopping. The same team sets both budgets in the same meeting, on the same day, out of the same pot. Over two years of weekly numbers the two spend columns correlate at about 0.9.
Least squares looks at that and returns plus one thousand on Search and minus nine hundred and ninety-eight on Shopping. Keep both numbers and the predictions are fine. The two monsters cancel and what survives is the two euros of real effect. Take either number out of the room on its own and it is nonsense. Nobody spends one euro on Search and gets a thousand back, and nobody has ever lost money by turning Shopping on.
So what would a reasonable person do, handed a pair like that? Probably say something like: whatever these two channels are doing, they are doing it together, so give them roughly the same number, and make both of those numbers small. That is not a fudge and it is not a compromise. It is the answer. Ridge is that sentence, written down so a computer can follow it.
Two sentences of recall from the first module and then we move on. Least squares aims at the truth on average, and pays for it by lurching a long way whenever the data moves a little. Ridge aims slightly off on purpose, and in exchange it stops flinching. A bit more shift bought a lot less spread.
The rule comes before the formula. From now on the model pays for the weights it uses. Square each weight, add the squares up, and that number goes on the bill. Big weights are expensive. Two medium weights cost less than one enormous one, because squaring punishes the extremes far harder than it punishes the middle. Two weights of one cost two. One weight of two costs four. Same effect on the prediction, twice the price.
Here is the whole argument, and it takes no calculus at all. In this brand Search and Shopping are the same column: five weeks, spend of 1, 2, 3, 4 and 5 in both, revenue of 2, 4, 6, 8 and 10. Any two weights that add up to two predict every week exactly. There are infinitely many such pairs and the data likes them all equally well. Least squares has nothing left to say. So ask the bill instead.
| Search | Shopping | Error | Sum of squares |
|---|---|---|---|
| 2 | 0 | 0 | 4 |
| 1.5 | 0.5 | 0 | 2.5 |
| 1 | 1 | 0 | 2 |
| 1000 | −998 | 0 | 996 004 |
Ridge picks one and one. Four numbers in a column, and the question that least squares could not answer is settled. The pair that splits the credit evenly is the cheapest pair on the list, and the pair that made the meeting unbearable costs about half a million.
When the columns do not overlap at all, the ridge answer has an embarrassing shape. Take the least squares weight. Divide it by one plus lambda. That is the entire calculation.
Every weight is scaled by the same factor, and that factor is smaller than one. Eight halved is four. Four halved is two. Two halved is one. One halved is a half. Halve it as often as you like and it never arrives.
You can meet this before any geometry, and you should, because the geometry is a second route to the same place rather than the proof. Here are four weights with the dial set at lambda equals two, so everything is divided by three.
| Weight before | Divided by 1 + λ = 3 | Exactly zero? |
|---|---|---|
| 8 | 2.67 | no |
| 3 | 1.00 | no |
| 1.5 | 0.50 | no |
| 0.4 | 0.13 | no |
One honest caveat, said once. That exact division holds when the columns do not overlap, and real columns always overlap, which is the entire reason you are reading this. The behaviour survives anyway. The demo below is a real fit on real overlapping columns, and nothing in it ever reaches zero either.
Five channels, two years of weekly revenue. Search and Shopping are near enough the same column: in this dataset they correlate at 0.999, which is what happens when one person sets both budgets. Meta, TikTok and email are ordinary, well behaved columns. At lambda equals zero you get least squares, and least squares does the thing from the cold open.
Drag the dial. Watch the two monsters come screaming in toward each other, watch every bar get shorter, and watch what happens at the far right end.
The left panel is drawn to a fixed scale, so you can see how much smaller everything gets. The right panel rescales itself every time you move the dial, so you can always see all five bars no matter how short they are. That right panel is the point of the whole module. Crank lambda to the maximum, look at it, and count the bars.
One thing worth naming, because you will see it happen. Shopping starts below the line at minus fourteen and ends above it. On the way up it passes through zero exactly once. Passing through is not the same as being set to zero and left there. Nudge the dial one step and Shopping is back. In the next module you will meet a penalty that puts a coefficient on zero and keeps it there for every larger lambda, and the difference between passing through and staying is the difference between the two methods.
Least squares has a formula, and inside that formula there is one step that has to be undone. Everything depends on the undoing being possible. The old word for a thing you can undo that way is regular. The word for a thing you cannot is singular.
When two of your columns say the same thing, that step is singular. Notice what least squares does then, because it is not what people expect. It does not return a wrong answer and it does not warn you. It returns infinitely many exactly equal answers, and your software hands you whichever one its algorithm reached first. Run it again with the columns in a different order and you can get a different one.
Here is the repair on four numbers instead of on matrices. Go back to the five weeks. Both columns are 1, 2, 3, 4, 5. Multiply the columns together pairwise and add up: Search with Search is 55, Search with Shopping is 55, Shopping with Search is 55, Shopping with Shopping is 55. That is your whole table.
| Search | Shopping | |
|---|---|---|
| Search | 55 | 55 |
| Shopping | 55 | 55 |
Cross-multiply and subtract: 55 times 55 minus 55 times 55 is zero. Zero is the symptom. Zero is what singular looks like when you write it out as numbers. Now add lambda equals one to the two cells on the diagonal, so the table reads 56, 55, 55, 56. Cross-multiply and subtract again: 3136 minus 3025 is 111. Not zero. And it will never be zero, for any lambda above zero, for any dataset anybody has ever collected.
You made a singular thing regular. That is the job the word regularisation is doing in the name of this class. It is a piece of nineteenth century matrix vocabulary, not a metaphor about a model learning modestly.
Look again at what the demo did to Search and Shopping. It did not choose one. It walked them toward each other until they were carrying roughly the same weight, and then it walked them down together. That is not a side effect. It falls straight out of charging for squares, and you proved it yourself in the four-row table: one and one is cheaper than two and zero.
When two channels really do work together, this is exactly what you want. The budget meeting sets them together, the customer sees them together, and a model that reports them together is telling the truth about how the business runs. Ridge will not hand you a story about Search being the hero and Shopping being dead weight, because it has no way to prefer that story and no reason to.
Now the other side, and it is the reason the next module exists. Ridge gives you every channel back, slightly quieter. Ninety-one columns in, ninety-one columns out, all of them small, none of them gone. If somebody asks which three channels to cut on Monday, ridge has no answer for them, and it never will, for the reason you already know: multiplication cannot produce a zero.
People call lambda a complexity dial and then never say what is being dialled. There is a real quantity underneath. Take the marketing model with ninety-one columns against a hundred and four weeks. Once the penalty is switched on, that model does not use ninety-one parameters. It uses a decimal number of them, and the number slides.
| λ | Parameters actually in use |
|---|---|
| 0 | 91.0 |
| 1 | 41.7 |
| 10 | 18.4 |
| 100 | 7.2 |
| 1 000 | 2.6 |
| → infinity | → 0 |
Nobody has ever had a tenth of a parameter and there it is anyway. Ninety-one columns and seven point two parameters in use is a model you could almost explain to somebody. Note the bottom row. The count runs down to zero, but it gets there smoothly, and no individual weight is zero anywhere on the way. Those are two different things and this table is where students usually confuse them.
Least squares has a famous property. Among all unbiased estimators it has the smallest variance there is. Generations of students learned that and stopped, because it sounds like the end of an argument. Nobody ever asked for unbiased. It is a technical word, not a compliment. Unbiased means that if you repeated the whole study across infinitely many parallel universes your average error would come out at zero, and that is a comfort nobody has ever needed on a Tuesday.
Hoerl and Kennard proved something better in 1970. There always exists a lambda greater than zero whose total error is lower than the total error of least squares. Not usually. Not for well behaved data. Always, for every dataset that has ever been collected and every one that ever will be. They could not tell you which lambda it is, and neither can we. It is there anyway, and finding it is what the test error curve in the demo is for.
If that sounds like something you have already met, it is. It is the regression version of Stein's paradox from the first module. Shrinking your estimates toward a common point beats leaving them where they landed, and the theorem does not care whether the things you are estimating have anything to do with each other. Same fact, two languages.
Two boxes. Skipping both costs you nothing you will be examined on.
Write down two things you already think. First, how noisy your data is: call that variance sigma squared. Second, how large you expect the real effects to be: call that variance tau squared. Lambda is the first divided by the second.
Noisy data and modest expected effects: big lambda, shrink hard. Clean data and one channel you genuinely expect to dominate: small lambda, trust the fit. In plain words, lambda is how much you believe your own two years of history against your prior belief that no single channel is a miracle. Both of those are opinions you already have.
Where it comes from, and here is a correction worth stating out loud, because plenty of courses and at least one table of contents get it wrong. Assume Gaussian noise on the data. Then assume a Gaussian prior on the weights, centred on zero, with variance tau squared. Ask for the most probable set of weights given the data, which is maximum a posteriori estimation. Out comes ridge exactly, with lambda equal to sigma squared over tau squared.
Gaussian noise plus maximum likelihood does not give you ridge. Maximum likelihood on its own gives you ordinary least squares and nothing else. The prior is where the shrinkage comes from, and the estimation has to be MAP rather than maximum likelihood. Swap the Gaussian prior for a Laplace prior and the same machinery hands you the penalty from the next module instead.
Ridge does not shrink every direction by the same amount. Break the data into its principal directions, with singular values \(d_j\) measuring how much the data varies along each one. Ridge multiplies the direction by
Large \(d_j\) and that fraction sits near one, so the direction is barely touched. Small \(d_j\) and the fraction drops toward zero, so the direction is crushed. Which is exactly right. A direction with a tiny singular value is a direction in which your data barely varies, and the direction in which Search and Shopping disagree is precisely that direction. Least squares will happily put an enormous number there, because out there a huge coefficient costs almost nothing in fit. Ridge shrinks hardest exactly where the data knows least.
Add those fractions up over all the directions and you get the effective degrees of freedom, which is the right hand column of the table above:
At lambda equals zero it equals the number of features. As lambda grows it slides continuously down toward zero. Continuously, with no steps and no gaps, which is what makes lambda a genuine dial rather than a switch. And note again that the count reaching zero and an individual weight reaching zero are different events. The first happens. The second does not.
Ridge handed you every channel back, slightly quieter. Nothing was deleted, nothing was accused, and the meeting is survivable. That is worth a lot and it is not everything. Sometimes what you actually need is somebody to tell you which three channels to cut.
That job needs a different penalty, and it behaves nothing like this one. Instead of dividing every weight, it subtracts a fixed amount from every weight and stops at zero. Subtraction can reach zero. Multiplication cannot. Same dial, same data, completely different answer, and a different meeting.