Neural networks without the mathematics

A unit weighs its inputs

A neural network is one kind of model, the name for a computer program that has learned from examples how to turn an input, such as an email, into an answer, such as whether the email is spam. The network is built from a very large number of simple parts called units, loosely inspired by the way brain cells pass signals to one another. Each unit does one small job: it takes in a few numbers, decides how much each one counts, adds them up and passes on a single number of its own.

How much each input counts is set by a weight, which works much like a volume dial on a sound engineer’s mixing desk. The unit adds up its inputs after each has passed through its dial, much as the desk blends its channels, each at its own volume, into one sound. A weight’s dial differs from a volume dial, since zero sits in the middle of its range and the dial can be turned down past it. Set at zero, the dial keeps its input out of the unit’s total altogether; turned up, it makes the input count towards the total, more the further it turns; turned down past zero, it makes the input count against the total in the same way.

A unit built by hand to judge whether an email is spam might, for instance, take in the number of links, whether the sender is known and how often the word “urgent” appears. Each input has to be a number, so the answer to whether the sender is known goes in as 1 if the sender is known and 0 if not. The person building the unit might set the dial on the links well above zero, the one on a known sender below zero and the one on “urgent” a little above zero. Many links would then push the unit’s total up, and a known sender would pull it down.

Each unit also has a starting score of its own, the figure its total starts from before any input is counted. The technical name for the starting score is bias, used in a sense unrelated to prejudice or unfairness. The spam unit might start below zero, for instance, so an email has to show some signs of spam before the total climbs above zero. Finally, the unit applies a simple fixed rule, such as turning any negative total into zero, so it passes on zero for an ordinary email and a larger number the stronger the signs of spam.

The spam unit is a simplification, since a person chose its inputs and set its dials. In a real network, the only inputs people choose are the raw material the network reads, such as an email turned into numbers. Every dial and starting score is set by training, an automatic process that adjusts them one example after another until the network gives good answers. Most units take their inputs from the outputs of other units, whose dials training also set, so a person did not choose what those outputs stand for. One unit in a spam filter might, for instance, come to respond to a mix of words common in scam emails, without a person having decided that it should.

Layers

Units are arranged in rows called layers, and every unit in a layer reads the same inputs, each through its own set of dials. The outputs of one layer become the inputs of the next, so a signal passes in turn from the first layer to the last. The first layer reads the raw material, turned into numbers, such as the brightness of each pixel in an image or a separate list of numbers for each piece of a sentence, and the last layer gives the answer.

Stacking layers lets a network build complicated judgements out of simple ones, much as a firm builds an opinion on a transaction from many small checks, each relying on the checks before it. Deep learning, the name for the approach behind modern AI, refers to networks with many layers stacked in this way. Depth, meaning the number of layers, has in practice been central to the networks that now recognise faces, transcribe speech and write text.

Dials set by training

Before training, every dial and starting score in a network is set at random, so the network’s answers start out as nonsense. Training then works through a large collection of examples one at a time, each an input paired with the answer the network should give, in the way described in How a machine learns from examples, a Beginner module. After each example, a calculation compares the network’s answer with the right one and works out which way each dial and starting score should move to bring the two a little closer, and every one of them is nudged a fraction in that direction. The calculation is called backpropagation, explained for readers who want the detail in How a model learns, an Intermediate module.

Picture a mixing desk with billions of dials and an assistant who holds a reference recording of how each song should sound. After playing each song through the desk, the assistant compares the result with the recording and nudges every dial a hair towards a closer match. The assistant applies the same fixed calculation to every dial, without any view about what a particular dial is for, and after millions of songs the desk comes close to the reference recordings. Each dial’s final setting is the sum of millions of tiny nudges, each made for a different song while billions of other dials moved too, so the reason a particular dial ended up where it did is usually impossible to state.

Individual dials rarely have a meaning a person could name, since what a network has learned is spread across a great many of them working together. Why a network gave a particular answer is therefore hard to work out from the dials, which leaves whoever has to explain one of its decisions with a difficulty examined in Looking inside and testing, an Advanced module.

One large file of numbers

The dial settings and starting scores are saved once training ends, and those saved numbers are together called the model’s parameters. Dial settings make up nearly all of them, so people often call the whole set the model’s weights. Everything else about the network, from the number of layers to the arithmetic each unit performs, stays fixed throughout training and makes up the model’s recipe. The recipe is set out in program code, meaning the instructions a computer follows, with the help of a short configuration file, which records sizes that training does not change, such as how many layers the network has. A trained model is therefore a short piece of program code and a configuration file, which set out the fixed recipe of arithmetic, together with one large file of numbers. The recipe is small and often public, while the file of numbers holds everything the model learned.

Meta’s public release of its model Llama 3.1 70B shows the difference in size between the recipe and the file of numbers. The model’s files on Hugging Face, a site that hosts models for download, hold about 70 billion weights in 30 files totalling about 141 gigabytes, in the format that most software for running models reads. The 30 files form one set of numbers, split into parts of up to five gigabytes for easier download. A separate folder in the same release, outside the 30 files, holds a second copy of the weights in Meta’s original format, also about 141 gigabytes.

The program code itself comes from widely used software for running models, so the release supplies only the part of the recipe specific to this model. The part specific to Llama is its configuration file, 855 bytes long, which tells the code how many layers to build and how large to make each one. The 141 gigabytes are the model, in the sense relevant to a business that grants a licence over it, copies it or loses it.

The file of weights is a distinct asset, separate from the code that runs it and from the data that shaped it, and each of the three can be licensed, leaked or sold on its own. Whether a file of weights can contain copies of the works it was trained on is a question courts have begun to consider, as set out in Where AI meets the law: a map, a Beginner module. Readers who want to go further will find the same ideas with the arithmetic written out in What a model is, an Intermediate module.