Lasso: Some Coefficients Die

Seven smaller numbers is not an answer

The marketing director has a budget meeting on Thursday and one decision to make. Spend is coming down by fifteen percent and three channels have to go. She has asked you which three.

You hand her the ridge output. Search 2.7, Shopping 1.8, Meta 1.9, TikTok 1.1, affiliate 0.6, display 0.3, email 0.3. Every number smaller than it was, every number stable, every number defensible. She reads it, and then she asks the question again.

Ridge cannot answer it. It has an opinion about all seven channels and it holds all seven opinions at once. Nothing in that list is zero, so nothing in that list is a name you can cross out. The method that refuses to pick a favourite also refuses to name a loser, and those turn out to be the same refusal.

What she wants is a shorter list. Not smaller numbers. Fewer of them.

Ridge multiplies, lasso subtracts

Three rules for what to do with one coefficient. Take the clean case first, where the columns do not overlap at all, because there the arithmetic finishes and you can check every step by hand.

Now say the difference out loud, because it needs no geometry and no picture. Eight halved is four. Four halved is two. Two halved is one. Halve it as often as you like and it never arrives. Multiplying by a number smaller than one shrinks forever without landing on zero.

Subtracting arrives. Take away a fixed amount often enough and you run out of coefficient.

One coefficient, three settings of the dial

A channel comes back with a coefficient of 0.30. Watch what each method reports as you turn \(\lambda\) up.

Ridge goes 0.27, then 0.23, then 0.20. It is heading somewhere and it is not getting there. Lasso goes 0.20, then zero, then zero. Once a coefficient has been pushed to the wall it stays at the wall for every larger \(\lambda\), so a channel that dies stays dead.

The rule written once, and you never need to write it again:

The little plus sign at the bottom right means "and if that came out negative, use zero instead". That is the entire wall.

One honest caveat, said once and not buried. This exact arithmetic holds when the columns do not overlap, and real columns always overlap, which is the reason you are reading this at all. It is still the right picture. The next section gives you a second and independent argument that covers the messy case.

The version that lives in your bank account: cut every subscription by a fifth and you are still subscribed to all of them, still paying, still annoyed. Take a flat five euros off each one and the ones you had forgotten about disappear.

One slider, two methods, one difference

Same seven channels as the ridge module, same 104 weeks, same correlations. Search and Shopping move together at 0.88 in this brand, because they are set in the same meeting by the same team.

One slider drives both panels. Drag it slowly and watch the left side and the right side stop agreeing.

Push the slider to the far right and the left panel still has seven bars. Small ones, but seven. The right panel has one, and the one it kept is Search.

The strip underneath is the part a marketing director can use. Lasso does not only produce a shorter list. It produces the list in an order, because the dial tells you which channel gave up first, which gave up second, and which was the last one standing. That is a ranked shortlist, and it fell out of a penalty rather than out of anybody's opinion.

Notice also what the left panel is doing to Search and Shopping. As \(\lambda\) rises, ridge pulls the two of them toward each other and hands them roughly equal credit. It cannot tell them apart, so it refuses to choose. Hold on to that, because in a moment it stops being a virtue.

A second door: the diamond has corners

What follows is a second explanation of the same fact, and it is independent of the first. If the arithmetic above slid past you, this one still works. If the picture below does nothing for you, the arithmetic above is already a complete argument. Nobody has to get through both doors.

Two coefficients, so the whole thing fits on a page. Instead of charging a price for size, put a budget on it. The coefficients have to fit inside the budget, and you take the best fit you can get inside it.

• All the pairs whose absolute values add up to the budget: that set is a diamond. This is lasso.
• All the pairs whose squares add up to the budget: that set is a circle. This is ridge.

The error surface is a bowl and its lowest point is the least squares answer. Slice the bowl and you get a closed curve of pairs that fit equally well, and for squared error that curve is an ellipse. Slice higher and the ellipse is bigger. So the ellipses grow outward from the least squares answer, and the first place one of them touches your budget region is the answer you get. Before it touches, nothing in the region fits that well. After it touches, you have left the budget.

A diamond has four corners and every corner sits exactly on an axis, which means one of the two coefficients is exactly zero there. A corner also sticks out, which is what makes it a corner, so an ellipse arriving from a generic direction meets a corner far more often than it meets any particular stretch of flat edge. A circle has no corners at all and no distinguished point anywhere on its boundary, so the touch point has both coordinates non-zero, and it does so essentially always.

Drag the least squares answer around and watch both touch points. The number under each panel is the second coefficient at the solution.

Here is Tibshirani in 1996, describing his own Figure 2. Lasso first:

And ridge, in the same passage, has

If you picked the diamond in the box above, you got there on your own, before reading it, and you agree with the man who invented the method.

He calls the diamond a square, and he is right: a square turned forty-five degrees is still a square. We quote him as he wrote it rather than quietly correcting him, because a book that improves its quotations is a book you cannot check.

And it scales, which is the part that matters, because your real problem has ninety-one coefficients and not two. In ninety-one dimensions the same shape has corners, edges and flat faces of every dimension from zero up to ninety. A corner is where ninety coefficients are zero. A flat face is where one is. Every subset of your channels has a piece of that boundary belonging to it, which is why lasso can zero out any subset at all.

The word came first, the acronym came second

Least Absolute Shrinkage and Selection Operator, says every textbook. The expansion was fitted to the word afterwards and Tibshirani has said so. He picked the word because a lasso is the thing you throw over something to drag it in. The paper is Tibshirani, "Regression Shrinkage and Selection via the Lasso", 1996.

It exists because the two options already on the table were both bad, each in its own direction.

• Subset selection gives you a model you can read out loud. It is also a discrete process: drop one observation, refit, and you can get a completely different set of variables. That is high variance in exactly the sense you learned to worry about.
• Ridge is stable. It never simplifies anything. You are still holding every column you started with.

The immediate precursor is Breiman's non-negative garrote, which starts from the least squares estimates and shrinks them by non-negative factors whose sum is capped. Tibshirani's stated improvement is that the lasso "avoids the explicit use of the OLS estimates". Read that with the first module in mind. Not leaning on the least squares estimates matters precisely when the least squares estimates are broken, which is the DuPont situation, which is your marketing meeting. The two halves of this class are historically connected and not merely next to each other.

The worked example in the paper is the prostate cancer data. Eight predictors go in. Three come out: lcavol, lweight and svi. Subset selection had picked the same three. Three is a sentence. Eight is a spreadsheet.

It dropped Shopping, and that was almost a coin flip

Now the bill, and do not let anybody soften this for you.

Give lasso two predictors that say nearly the same thing and it keeps one and sets the other to exactly zero. Which one it keeps is close to arbitrary, because the two answers score almost the same, and a near tie gets broken by whatever noise happened to be in the window. Search and Shopping correlate at about 0.9 in this brand. Same team, same budget, same seasonal push.

Slide the two-year window forward by a few months, refit, and the model makes the opposite choice with exactly the same apparent confidence. Nothing about the business changed. The tie-break changed.

And "the model dropped Shopping" is a sentence somebody will act on. It is a budget line, a quarterly target, a team, possibly a job. A number that came from a tie-break gets read in the room as a finding.

That instability is real and it is the reason elastic net exists. Charge for the absolute values and for the squares at the same time, with a dial between them. The diamond keeps its corners so you still get zeros, and the corners get less pointy, so correlated predictors enter and leave together instead of one of them being deleted by a rounding error. That last property has a name, the grouping effect, and it is the only thing you need to remember about elastic net today.

Which leaves the sentence this section exists for, and it is not a technical sentence. Lasso says drop TikTok, which is a budget line somebody loses and a meeting somebody has to sit through. Ridge says keep everything and trust nothing individually, which is a different meeting with a different outcome. Choosing between the two is not a modelling preference.

Ninety percent of random gene lists worked

You have far more columns than rows. So does cancer genomics, by a much wider margin, and it has already run the experiment you would not be allowed to run.

When you have far more columns than rows, "my model found a signature and it validated" is nearly worthless as evidence, because a list of randomly drawn genes validates too.

A test that almost everything passes carries no information, and you cannot see that from the inside. It only becomes visible when somebody feeds the test nonsense on purpose and the nonsense passes. So when you are told that a model was validated, ask what would have failed.

Real patients were treated on the basis of lists like these.

Where this leaves you

Three modules in, you have a dial that trades a little shift for a lot less spread, and two shapes of penalty that behave differently at the same setting of it. Squares shrink everything and keep everything. Absolute values shrink everything and then start deleting, in an order you can read off the slider.

Now look at what you actually did to choose \(\lambda\).