Every exam question is already published. For this course, learning them cold is the goal. For a machine learning model, it is the trap. The train/test split is how you tell the two apart.
Every question that can show up on the final is already published. The full pool, nothing hidden, sitting on one page: the exam pool.
Go memorize them. For this course that is not cheating, it is the point. This is a foundations course. The basics, what a residual is, what the confusion matrix counts, what overfitting looks like, should sit under your fingertips, recallable cold. Drilling the exact questions is a fine way to get them there.
So the usual warning, do not just memorize, does not apply to you here. It applies to the thing you are about to build.
You have the full pool of questions. Pick how you would actually study for the final.
Earlier in the course we gave memorizing a bad name. A model that overfits has memorized its training data instead of learning the pattern behind it. It scores beautifully on the rows it has seen and falls apart on anything new.
You do not want a churn model that has memorized last quarter’s customers. Those customers are gone. You want one that learned the shape of churn, so it works on customers it has never met. Memorizing the training rows and generalizing to new ones look identical until you put the model in front of something new.
The train/test split is how you force the question. Train on some of the data. Hold the rest back, sealed, untouched. Then score the model on the sealed part. If it learned the pattern, it does well on data it never trained on. If it only memorized, the sealed score exposes it.
The score on the sealed part is an estimate. You did not test on every possible question, only on the few you set aside. So the number carries error, the same way a poll of ten people carries more error than a poll of ten thousand.
If your true chance of getting a question right is \(p\), and you test on \(n\) sealed questions, the spread of your measured score is the standard error:
Two things to read off it. The \(n\) is under a square root, so to halve your uncertainty you need four times as many test questions, not twice. And the \(p(1-p)\) on top says the spread depends on how variable the outcomes are. A model that is right almost always, or wrong almost always, barely varies, so a small test set already pins it down. A model near a coin flip is as uncertain as it gets and needs far more.
To pin accuracy to within about ten points, a coin-flip-hard problem needs roughly a hundred test questions. A model that is genuinely 95% accurate needs about twenty. Same target, very different test sets, because the precision rides on the variance of the outcomes, not on any fixed percentage.
All of this holds only if the sealed questions came from the same pile as the real ones. Seal a hundred questions on French history, then sit a final on organic chemistry, and your confident sealed score predicts nothing.
A test set is the same. It estimates how the model behaves on new data only when the new data looks like the test data. Train a churn model on last year’s customers, deploy it on this year’s, and if this year’s customers behave differently, the test score was a confident lie.
Two sliders. The first sets how you split the hundred questions between training and test. The second sets how predictable the material is, which is the part people forget. Watch the accuracy the model reaches, the accuracy the test set reports, and the test set it would actually take to trust that number.
The test set hands you a number somewhere inside the shaded band. You do not get to know where. More sealed questions narrow the band. A more predictable problem narrows it too, for free.
Notice the 80/20 mark. Eighty percent training, twenty percent test is what everyone reaches for, and it is a habit, not a result. It fixes a ratio and ignores the two things that actually decide your test set: how many examples you have in total, and how variable the outcomes are. Twenty percent of fifty rows is ten test examples, far too few to trust. Twenty percent of a million is two hundred thousand, far more than you will ever need. A predictable problem lets you seal fewer and pour the rest into training. The right split comes from the data, not from a number someone repeated.
The required test size for a fixed precision comes straight from the standard error. For a 95% interval of half-width \(w\), you need about:
Set \(w\) to ten points. A coin-flip problem at \(p = 0.5\) needs about 97 test examples. A 90% model needs about 35. A 95% model needs about 19. A 99% model needs about 4. The same precision, costing wildly different amounts, all driven by that \(p(1-p)\) term. This is the math behind every accuracy figure you read. A model reported at 95% on 20 test examples is barely making a claim. The same 95% on ten thousand is a real one.
You have 100 labeled customers and want to predict who will churn. You set aside 10 as a sealed test set, train on the other 90, and the model scores 95% on the 10. A colleague sets aside 50, trains on 50, and scores 88% on their 50.
Whose number would you put in a report to the marketing team, and why? And which of the two numbers is the more dangerous one to repeat out loud?
Splitting once forces a choice. A bigger training set or a bigger test set, never both from the same data. Cross-validation is the trick that lets you stop choosing, by sealing a different part every time and averaging the results. That is the next module.