A conceptual introduction, from a single neuron to deep learning
A single neuron is almost embarrassingly simple. It takes a few numbers in. It multiplies each one by a weight. It adds them up, adds one more number called a bias, and passes the total through a function that squashes it into a clean output.
Say you are deciding whether to send a customer a win-back coupon. You look at a few things. Days since their last order. How many support tickets they opened. Whether they still open your emails. Each of those gets a weight, depending on how much you think it matters. You add up the weighted evidence. If the total clears a threshold, you send the coupon. If not, you do not. That is a neuron.
The weights decide what the neuron pays attention to. The bias decides how easily it fires. The function at the end, often the sigmoid, turns the raw total into a number between 0 and 1, a kind of confidence.
If this feels familiar, it should. One neuron with a sigmoid on the end is logistic regression from Class 6, wearing a different hat.
One neuron makes one decision. Put a row of them side by side, each looking at the same inputs but with its own weights, and you have a layer. One neuron might learn to watch for "has not ordered in a while." Another watches for "still reading the emails." Each one becomes sensitive to a different pattern in the same data.
Now take the outputs of that layer and feed them as inputs to a second layer. The second layer never sees the raw customer data. It only sees what the first layer made of it. A third layer sees what the second made of that. Each layer works on the layer below, building on what came before. This is what people mean when they say a network composes transformations.
The squashing function is doing quiet, heavy work here. Take it out and something embarrassing happens. A stack of plain weighted sums, one feeding the next, collapses back into a single weighted sum. Ten thousand linear layers are still one straight line. The bend in the activation function is the only reason depth buys you anything at all.
Show a neural network a million photos, told only which ones contain a cat and which do not. Watch what the layers learn. The first layer learns to notice edges, small slants of light against dark. The next layer puts edges together into corners and curves. The one above that assembles curves into eyes, ears, the texture of fur. By the top, some neuron has quietly become a cat detector.
Nobody wrote "look for ears." Nobody wrote "fur goes here." The only instruction was the loop you already know from linear regression. Make a guess, measure how wrong it is, nudge every weight a little in the direction that lowers the error, repeat a few million times.
That is the whole idea behind deep learning. Deep means many layers stacked up. Learning means the weights were tuned by that loop, not set by a person. The useful structure, the edge finders and the cat finders, falls out of the training on its own.
In 1958, Frank Rosenblatt built a machine called the Perceptron at the Cornell Aeronautical Laboratory. It was a single layer of these units wired to a grid of light sensors, and it learned to tell simple shapes apart by adjusting the strength of its connections. The press got carried away and wrote about machines that would soon walk and talk.
Then in 1969, Minsky and Papert proved a single layer could not even learn exclusive-or, the simplest "one or the other but not both" rule. Funding dried up. The fix came later. Stack a hidden layer in the middle, and send the error backwards through the network to decide how much each weight was to blame. That trick is backpropagation, popularised by Rumelhart, Hinton, and Williams in 1986, and it is still how almost every network learns today.
Press play. The signal runs forward to make a guess, the guess is compared to the right answer, and the error walks back through the weights to correct them. One pass of that is one step of training. Real networks run it millions of times.
The activation function is the squash at the end of each neuron, the bend that keeps a deep stack from collapsing into one line. The classic one is the sigmoid, which maps any number into the range between 0 and 1:
It has a problem. For large positive or large negative inputs, the curve goes flat, and a flat curve has almost no slope. Gradient descent learns from slope. When the slope vanishes, the backward pass carries almost nothing, and learning stalls. Tanh has the same shape but runs from minus 1 to 1 and sits centred on zero, which helps a little:
The function that took over modern networks is plainer than both. ReLU. It returns zero for anything negative and passes positive numbers through untouched:
It is the cheapest of the three to compute, and its slope on the positive side never flattens, so the error keeps flowing back through deep stacks instead of dying out. That is most of why deep networks became trainable in the first place.