The simplest model of choices
Imagine a representative in Bundestag making decision to vote on a bill and doing mental maths on it. The bill aligns with the party platform: plus five toward yes. The bill conflicts with a public statement from three years ago: minus three and a half. The bill benefits the home district: plus two and a half. Each factor contributes to the final decision. Each factor has a fixed score.
The sum of the scores is a real number that can land anywhere on the number line. That sum is the log-odds. The final probability has to live in the interval zero to one. The "sigmoid" is the function that maps any real number onto [0,1] interval without ever quite reaching the edges. Log-odds zero comes out as exactly fifty percent. Log-odds plus three lands at about ninety-five percent yes. Log-odds minus three at about five percent. The curve approaches one and zero without touching them, which is the polite way of saying the representative is never entirely sure.
The whole thing fits in one line.
The sigmoid did not begin life as a probability tool. In 1845 the Belgian mathematician Pierre François Verhulst was studying how populations grow when resources are finite. Exponential growth is the early phase, when nothing is scarce. Saturation is the late phase, when everything is. The S-curve bridges them. He called it the logistique, from logarithm and arithmetic. It described rabbits, bacteria, and the population of Belgium itself.
A century later, in 1944, the American statistician Joseph Berkson was running bioassays. He needed to model the probability that a given dose of insecticide would kill an insect. The probability had to live in zero to one. He took Verhulst's S-curve, took the logarithm of the odds ratio, and named the result the logit. Same curve. Different problem.
Two features, two classes, one boundary. Imagine the first feature is how strongly a bill aligns with a representative's party platform. The second feature is how much it benefits their home district. Each dot on the plane is a past bill on which the representative voted. Green dots voted yes. Red dots voted no. The boundary is the line where the representative is fifty-fifty. On one side, mostly yes votes. On the other, mostly no.
Let's visualise every score in the mental model. Toggle a feature on, and its coefficient enters the sum. Toggle it off, and it abstains. The intercept votes whether you do or not, which is the polite way of saying the representative has a baseline lean even before they know anything specific about the bill.
The running total above is the log-odds. The curve in the middle is the sigmoid. The dot on the curve is where the internal Bundestag has landed. The probability on the right is the chance this representative votes yes.
| Feature of the bill | Coefficient | Vote |
|---|
A bill comes to the floor tomorrow. Imagine we know how the representative voted on sixty previous bills. We can run the thought process above, but build a logistic regression, fit it against their voting records on past bills of this kind, and depending on the bill calculate a particular probability to vote yes. If we want to predict the vote we can change the threshold of our model depending on our needs either making it skew towards predicting a yes vote (with lower threshold) or towards predicting a no vote (with higher threshold).
Drag the threshold. Everything to the right of the line is predicted as a yes vote. Everything to the left is predicted as a no. Watch the four cells of the confusion matrix update. Watch precision and recall fight each other. Watch the point move along the precision-recall curve.
The model produces a probability. The training data has a true label, zero or one. The loss says how wrong the model was. Squared error would have worked, mathematically. Cross-entropy is what everyone uses. The reason fits in one picture.
Imagine a single representative who in fact voted yes. The true label is one. The model produces a probability p, its estimate that this representative would vote yes. The plot below shows the loss as a function of p, under two different rules. The grey curve is squared error: (1 − p)². The amber curve is cross-entropy: −log(p).
When p is close to one, both losses are small. When p is around a half, both losses are middling. When p is very close to zero, the difference becomes brutal. Squared error caps out at one. The model was as confidently wrong as it could be, and it paid a price of one. Cross-entropy keeps climbing. A model that says "five percent chance of voting yes" about a representative who in fact voted yes pays a loss of three. A model that says "one percent" pays four point six. A model that says "zero point one percent" pays seven.
That is the point. If a model is going to call itself a probability, it has to be punished for being confident about wrong answers. Squared error does not punish enough. Cross-entropy does. The model learns to be uncertain when it should be.
Most of the decisions we make could be represented as a sequence of yes or no answers that we take with some probabilities depending on our previous experience. We can model customer behaviour choosing to buy something or click on a link. We can model if the letter is spam or not. If the transaction is fraudulent or real. The logistic regression is not the best classification model (as well as linear regression is not the best regression one), but it's a simple solid baseline that is very easy to interpret.