A language model has read everything and understood nothing. Watch it finish your sentence anyway.
Every model in this course had an honest job. Predict a price. Sort mail into spam and not spam. Draw a line, split a tree, count the neighbors. In each case you could say, out loud, what the right answer was supposed to be.
This one has a stranger job. Look at some words. Guess the word that comes next.
You already do this. You read three words of a text message and your thumb knows the fourth. Someone starts a sentence and you have finished it in your head before they reach the end. You are right often enough that you stop noticing you do it.
Nobody sat you down with a list of every sentence in the language. You heard it. Years of it, from the people around you, and somewhere in there you started completing sentences you had never heard before. You learned the shape of the language from examples, then used that shape to fill gaps that were new. Hold on to that. It comes back at the end.
Before the machine tries, try it yourself.
Nobody hesitates over that one. Now the same game, with less to go on.
The first gap had a winner. The second had a crowd. Same task both times, guess the next word, and the only thing that changed was how much context you were handed.
A machine that must answer every time cannot pick one word and commit. It spreads its bet. It hands back a probability for every word it knows. When the context is thin the probability spreads wide. When the context is tight it piles onto one word. Guessing the next word means producing that spread, every time, at every position.
So where does the spread come from? The dull answer, and the true one, is that the machine counts.
We have 1.3 million words of Reuters news wire, about 78 thousand sentences of takeovers, oil prices, and quarterly dividends. We read all of it and counted. For every pair of words, we wrote down which word came after it, and how many times.
That table of counts is the model. There is nothing else inside it. No idea what a company is, no notion of oil, no sense that a sentence points at anything in the world. It read the whole corpus and understood not one word. Ask it what follows the company and it does not think. It finds the row and reads off the tallies: said, 1,181 times.
The model stores counts. To turn counts into probabilities you could divide each by the total, and at temperature 1 that is close to what happens. Temperature is the knob that reshapes the spread before the model samples from it.
Let \(c_i\) be the count of candidate word \(i\), and set its score to \(s_i = \log c_i\). The probability of picking word \(i\) is:
Low temperature divides the scores by a small number, which stretches the gaps between them. The largest score pulls away from the rest and its probability climbs toward 1. The model turns greedy and keeps picking the top word. High temperature divides by a large number, which squashes the gaps. The scores drift together and the probabilities flatten toward uniform. The model turns reckless and will say almost anything.
Real language models do the same thing at the final step. They produce a score for every word in their vocabulary, divide by temperature, and normalize with that same formula. The scores come from a trained network instead of a shelf of counts, but the softmax at the end is the arithmetic you are dragging that slider through.
This counting model has a name. It is a Markov chain. The rule of a Markov chain is short. To guess what comes next, look only at where you are now, not at how you got there. For our model, where you are now is the last word or two. Everything earlier in the sentence gets thrown away before it guesses.
That forgetting is the whole problem. Read this. "The deal, which the board had fought over for months, finally ___." You feel the ending leaning on the deal and fought over, words far to the left. A model that keeps only the last two words never sees them. It guesses from finally alone.
The obvious fix is to keep more context. Hold the last five words instead of two. Here the counting hits a wall. This corpus has about 29 thousand distinct words. A table keyed on two words needs up to 29,000 squared rows. Keyed on five, it needs 29,000 to the fifth power, a 2 followed by 22 zeros, thousands of times more rows than there are grains of sand on Earth. Almost every row sits empty, because that exact five-word run never appeared. Counting cannot buy you a longer memory. The table explodes and stays blank.
Remember how you learned the language. Not from a table of every sentence, but from exposure, until you could finish sentences you had never heard. A neural network pulls the same trick. Instead of a row for every exact context, it turns the context into a short list of numbers, a vector. Contexts that behave alike land near each other in that space, so the company said it and the firm announced it sit as near neighbors even if the network only ever saw one of them. It fills the gap from the vector, not from a lookup.
You have watched this move before. In the recommendation module we started with a giant sparse table, one column per product, almost all zeros, and past a certain size it stopped working. The fix was to compress each user into a short dense vector of tastes. Same fix here. A dense vector for the context replaces a sparse table of every co-occurrence. Counting asks whether it has seen these exact words together. The network asks what this context resembles.
Type one sentence you are sure a business newspaper has printed a thousand times. Then type one you invented on the spot. Watch which one the model has an answer for, and which one collapses to the, of, to. The distance between those two is the problem the rest of this course is about.