How a model learns

Measuring error

Training sets a model’s weights, the numbers that turn its input into its output, by starting from random values and adjusting them, example by example, until the model’s outputs come close to the answers in its training data. In supervised learning, each example comes with its correct answer attached by a person, such as a photograph tagged “cat”, and that answer is called a label.

A language model, the kind of model behind chatbots such as ChatGPT, is trained mainly to predict the next token of a text, a token being a word or a piece of a word. Pretraining, the main stage of a language model’s training, uses the text itself as the answer key: the correct answer at each point is the token that comes next, so the text supplies its own labels, an arrangement researchers call self-supervised learning. Labels written by people come in only at the later stages, which train the model on example answers and on people’s preferences to turn it into a helpful assistant. Self-supervised learning lets a developer train on huge quantities of text gathered from the web without paying people to label it. The collected text is therefore the raw material of training, which is why legal questions about training so often start with where that text came from.

Training also needs a score for how wrong each answer was, and a loss function supplies one by turning the error into a single number, called the loss, which training then tries to push down. A language model gives a probability to each possible next token, and the standard loss for models that work this way, called cross-entropy, charges a small penalty when the model gave the right answer a high probability and a steep one when it gave that answer a low probability. Giving the right answer a probability of 90 per cent costs about 0.1, giving it 10 per cent costs about 2.3, and giving it 1 per cent costs about 4.6. A low probability for the right answer means the model put its confidence in wrong answers, so a confident mistake is punished far more heavily than a hesitant one. During its main training a language model is scored only on how well it predicts the text in its data, so training pushes it towards output that reads like that text, whether or not what it says is true.

Walking downhill

With the error measured, training has to work out which way to move each weight, and a calculation called the gradient gives that direction for every weight at once. The gradient is the slope of the loss: for each weight, it shows how fast the loss would rise or fall if that weight were nudged up a little, measured as the change in the loss divided by the size of the nudge. Going downhill therefore means moving each weight in the direction that lowers the loss. Gradient descent, the method behind the training of nearly every large AI model, moves each weight a small step downhill, measures the slope again and repeats, and a number called the learning rate sets the size of each step.

Consider a model with a single weight w that multiplies its input by w, trained on one example whose input is 2 and whose correct answer is 6, so the ideal weight is 3. Starting at w = 1, the model predicts 2 and misses by 4, and the loss here is the miss multiplied by itself, 16, which counts a miss in either direction as positive. The slope can be found by nudging w: raising it from 1 to 1.01 lowers the loss from 16 to about 15.84, a fall of about 0.16, and dividing that fall by the nudge of 0.01 gives a slope of −16. Calculus gives the same answer directly, and the minus sign means that raising w will reduce the loss, while the match with the loss of 16 is a coincidence of these numbers.

Each step subtracts the learning rate times the slope from the weight, and since the slope here is negative, the subtraction makes the weight larger. With a learning rate of 0.05 the weight therefore rises by 0.05 times 16, which is 0.8, to 1.8, then to 2.28 and 2.568, closing in on 3 with each step.

Steps that are too large overshoot the target, and at a learning rate of 0.2 the weight lands on alternate sides of 3, going from 1 to 4.2 and then to 2.28, though it still homes in because each overshoot is smaller than the last. At 0.25 the weight jumps from 1 to 5 and back to 1 indefinitely. At 0.3 each step overshoots by more than the last, so the loss grows without limit. Real models take much smaller steps: the largest version of GPT-3, an OpenAI model in the line that led to ChatGPT, used a learning rate of 0.00006, a tiny step repeated many thousands of times.

A real model trains on a vast number of examples, and the slope it follows is that of the loss averaged across all of them. Measuring that slope over the whole dataset before every step would be far too slow, so each step uses a small random sample of examples called a mini-batch, which gives a rough but cheap reading of the downhill direction. Each of these steps adjusts every weight once and is called an update, and GPT-3 made on the order of 100,000 updates in its training. One full pass over a dataset is called an epoch, and a large language model often makes less than one pass over its largest sources, which means some of their text is not shown to the model at all. GPT-3, for example, saw about 44 per cent of its main collection of web pages and read Wikipedia about 3.4 times over. Developers choose how often the model sees each source, showing it the sources they rate more highly several times over, and the more often a model sees a passage, the more likely it is to memorise the passage word for word.

Gradient descent is usually pictured as a walker in fog who can feel only the slope underfoot and keeps stepping downhill, with height standing for the loss. The picture captures how the method works from the local slope alone, without any map of the ground ahead. The image misleads in other respects, since a position on a hillside is fixed by two numbers, such as a map’s grid reference, while a position for a model takes one number for every weight, billions in all. The ground also shifts slightly from one step to the next, since each step measures the slope on a different mini-batch of examples. The walker’s destination differs too, since the height of the ground is the loss on the training examples alone. Training aims for weights that perform well on data the model has not seen, and the very lowest point for the training examples can be one where the model fits the quirks of those examples so closely that it does worse on new ones.

The weights a model ends with are wherever this long series of small steps leaves them, since the developer does not set any of them by hand. A developer asked why a model gave a particular answer therefore does not have a written rule inside the model to point to, which is part of why explaining an AI system’s output to a client or a court is often hard.

Backpropagation

Large models are neural networks, built as a series of layers in which each layer’s calculation feeds the next, with billions of weights spread across them. Gradient descent needs the slope for every one of those weights at every step, and backpropagation is the calculation that works out all of those slopes in a single sweep. The model first makes its prediction in a forward pass, from the first layer to the last, and backpropagation then runs a backward pass from the error to the first layer. Each layer receives its share of the blame for the miss on the way back, so every weight learns which way to move and by how much.

Backpropagation resembles tracing a mistake in a report back through the people who worked on it, where the final error is passed back person by person and each contributor’s share depends on how much their work shaped everything after it. Calculus supplies the rule that makes the tracing possible, called the chain rule, under which a weight’s effect on the final error is its effect on the next stage of the calculation, multiplied by that stage’s effect on the stage after it, and so on to the end. Each layer holds many separate calculations side by side, so one weight’s result can feed several calculations in the next layer and affect the error by more than one route. The chain rule then adds together the effects along all of those routes.

Backpropagation became the standard way to train neural networks after a 1986 paper in Nature showed that it lets the inner layers of a network learn to detect useful patterns of their own.

Modern training uses a refinement of gradient descent called Adam, which sets the step for each weight from that weight’s own recent slopes. Adam keeps two running averages for each weight, one of its recent slopes with their signs, so that slopes pointing in opposite directions cancel out, and one of how large those slopes have been whatever their direction. Adam moves each weight by the learning rate times the first average divided by the second. A weight whose slopes keep pointing the same way therefore gets a ratio close to one, and so a full-sized step, however steep or gentle its slopes are, since the division cancels out their size. A weight whose slope keeps changing direction gets a much smaller step, since its opposing slopes cancel out in the first average while the second stays large. Adam, or a close variant of it, is the name a reader is likely to see in the training section of a developer’s technical report on a model.

Learning versus memorising

The aim of training is generalisation, meaning good performance on inputs the model has not seen. The failure to avoid is overfitting, in which a model fits the quirks of its training examples so closely that it does worse on new data, much as a student who memorises last year’s exam answers can come unstuck on new questions. Developers therefore split their data into a training set that adjusts the weights, a validation set used during development, and a test set held back for an honest final score. The validation set helps the developer choose the settings it does pick by hand, such as the learning rate and how long to train, and it shows overfitting starting when its score worsens while the training score keeps improving.

A test score is honest only if the test questions stayed out of the training data, which is hard to guarantee when that data is text collected from across the web. The resulting risk is called contamination, in which the questions from a benchmark, a standard published test used to compare models, leak into the training data and inflate the model’s score on that test. Contamination is one reason published benchmark results deserve caution, as Looking inside and testing, an Advanced module, explains, and a lawyer reviewing a supplier’s claims about a model’s accuracy can fairly ask how its test data was kept apart from its training data.

Classical statistics predicted that a model with more weights than training examples would memorise its data and fail on new inputs, since a model with that many adjustable numbers can match every training example exactly without picking up a general pattern. Image-recognition networks with far more weights than training photographs put that prediction to the test. The networks turned out to be able to learn the label of every photograph in a set whose labels had been assigned at random, which shows that they can memorise pure noise, data without any pattern to learn. The same networks generalise well when the labels are real, and why networks with far more weights than examples manage to generalise remains an open research question.

Generalisation and memorisation coexist in the same weights, and language models show it: researchers have extracted hundreds of passages word for word from GPT-2, an earlier OpenAI model, including people’s names, phone numbers and computer code. Later work found that memorisation grows with the size of the model, with the number of times a passage appears in the training data, and with the length of the opening text, or prompt, that researchers give the model to draw the passage out. Memorisation is how a protected work or a person’s details in the training data can come back out of a model word for word, so it bears on the copyright and data protection questions that training raises.

Chips and the two phases

Nearly all the work in a neural network is multiplication and addition on large grids of numbers, and since the sums within a layer do not depend on one another, a chip can work on thousands of them at the same time. A graphics processing unit (GPU), originally built to colour millions of pixels at once for video games, has thousands of simple arithmetic units suited to that work, and early experiments found it trained networks up to 70 times faster than a conventional processor. Training a large model keeps thousands of these chips busy for weeks or months, which is why the largest training runs cost so much.

The computing work a training run uses, which engineers call its compute, is counted in floating-point operations (FLOPs), each one a single multiplication or addition on numbers that have fractional parts, such as 0.5 or −1.3. The EU AI Act states its compute threshold for general-purpose models as a number of these operations, so a lawyer advising on that threshold needs a sense of how they are counted.

Weights are also called parameters, and a widely used rule of thumb puts training at about six operations for every parameter for every token the model reads. Roughly two of the six make the prediction in the forward pass, and the other four work out, in the backward pass, how each weight should change. Training compute is therefore about six times the number of parameters times the number of training tokens, written 6 × parameters × tokens. The rule checks out against a published figure, since Meta reported 3.8 × 10²⁵ FLOPs, which is 38 followed by 24 zeros, for its Llama 3 405B model. That model has 405 billion parameters and was trained on 15.6 trillion tokens, and six times those two numbers gives the same total. GPT-3, four years older, used about 3.14 × 10²³ FLOPs, roughly 120 times less. Developers have kept raising training compute partly because of the scaling laws, researchers’ finding that a model’s error tends to fall in a predictable way as the compute used to train it grows.

Training and inference are the two phases of a model’s life: training sets the weights once, at great cost, and inference, the phase in which the finished model answers queries, uses those weights unchanged. Inference costs roughly two operations per weight for each token the model reads or writes, so a single answer costs a minute fraction of the training run, although the total across millions of users becomes a large ongoing expense.

The weights stay fixed while a person chats with a deployed model, and a chatbot appears to remember earlier turns only because the whole conversation is fed back in as input each time. A provider that wants to learn from conversations has to collect them and use them in a later training run, which raises questions of data governance and of the terms the user accepted. The later run can be fine-tuning, the further training of an existing model on a smaller, targeted set of data, or part of the training of the provider’s next model.