What a model is

The mathematics used in this course

Nearly every calculation inside a model is multiplication and addition, repeated an enormous number of times, with a few simple rules such as the activation function applied in between. A reader who can follow a weighted sum, a list of numbers and a power of ten can therefore follow the rest of the course.

A file of numbers and a recipe

A trained model has two parts that only work together, an architecture and a file of parameters. The architecture is a short piece of program code that sets out a fixed recipe of arithmetic, multiplying numbers, adding them up and applying simple rules, layer after layer. The parameters, also called weights, are the numbers the recipe uses, and a large model has hundreds of billions of them, each one set during training. The same code loaded with a different file of weights behaves as a different system, in the way the same spreadsheet formulas give different answers when the fixed values in the cells they refer to are changed.

Meta’s public release of Llama 3.1 70B, a model with 70 billion parameters, makes the parts visible on disk. The weights sit in 30 files totalling about 141 gigabytes, saved in safetensors, the file format most tools load, with a second copy in Meta’s own format alongside them. The configuration file recording the layer sizes is 855 bytes long, and a separate tokeniser file of about 9 megabytes converts text into numbers in the way Module 3 describes.

A computer stores data in bytes, and a byte holds eight bits, each bit being a single 0 or 1. Each weight stored at 16-bit precision, meaning 16 bits for each number, takes two bytes, so 70 billion weights need about 140 gigabytes, a gigabyte being a billion bytes. Quantisation stores each weight with fewer bits after training, which shrinks the same model to about 35 gigabytes at four bits per weight, at some cost in accuracy that is usually small at eight bits and more noticeable at very low bit counts.

The training data is used to set the weights and is not shipped inside the weight file. Llama 3 405B was trained on about 15.6 trillion tokens, the fragments of words that Module 3 explains, which is very roughly 60 terabytes of text at an estimated four bytes or so per token, a terabyte being a thousand gigabytes. Its weights take about 810 gigabytes, around 74 times less than the text that shaped them, which makes the file far too small to hold a verbatim copy of its whole training set. Module 6 shows that a model can still memorise particular passages, sometimes at length.

The artificial neuron

The basic building block of a neural network is a unit, which popular accounts call a neuron. A unit takes a few input numbers, multiplies each one by its own weight, adds up the results together with one extra number called the bias, and then applies a fixed rule called an activation function to the total. A weight says how much its input counts, and a negative weight counts against the output, while the bias acts as a starting score that moves the point at which the unit switches on.

With inputs of 2 and 3, weights of 0.5 and −1 and a bias of 1, the first input contributes 0.5 × 2 = 1, the second contributes −1 × 3 = −3, and the bias adds 1, for a total of −1. A widely used activation function called ReLU keeps a positive total as it is and turns a negative total into zero, so the unit outputs 0. Current language models mostly use smoother relatives of ReLU, such as GELU and SwiGLU, which work on a similar principle.

An older activation function called sigmoid squeezes any total into a value between 0 and 1 along a smooth S-shaped curve, and it gives 0.269 for a total of −1.

A unit resembles a committee member who weighs several pieces of evidence, some for and some against, adds a personal predisposition, and speaks up only if the total clears a bar. The comparison fails in one respect that later modules return to, since the evidence reaching a unit deep inside a network is the output of other units, with a meaning that emerged during training and is usually hard for a person to read.

Layers and depth

A layer is a row of units that all read the same inputs, each unit with its own weights, and the outputs of one layer become the inputs of the next. The weights of a whole layer are stored as a matrix, a grid of numbers, so running a layer is one large block of multiplication and addition, which is the job the chips described in Module 2 are built for. Depth means the number of layers stacked in sequence, and it has grown from the eight learned layers of AlexNet, the image model that won a landmark competition in 2012, to 126 layers in Llama 3 405B.

The activation function earns its place by bending the arithmetic, since a layer that only multiplies and adds draws a straight-line relationship between its inputs and its outputs. Two straight-line rules applied one after the other still give a straight line, so without the bends a stack of a hundred layers would be limited to what a single layer can do. Each ReLU adds a kink, and many kinks joined together can follow a curve as closely as needed, which is how a network comes to represent complicated relationships.

Mathematicians proved in 1989 that a network with a single hidden layer, meaning one layer between the input and the output, can imitate any continuous relationship between inputs and outputs as closely as desired, provided it has enough units. A continuous relationship here means one without sudden jumps, and the proof covers a limited range of inputs. The proofs came from George Cybenko and, separately, from Kurt Hornik and his co-authors, and covered S-shaped activation functions such as the sigmoid. A 1993 result by Moshe Leshno and colleagues extended them to ReLU and to almost any activation function that is not a polynomial. The finding is often overstated, since it shows only that suitable weights exist, leaving open whether training will find them, how many units would be needed and how the network behaves on inputs outside the range it was fitted on.

What a parameter is

A parameter is closer to the strength of one connection between units than to a unit itself, so comparisons between parameter counts and the brain’s neurons tend to mislead. AlexNet, for instance, had about 60 million parameters spread across about 650,000 units, roughly 90 parameters for each unit.

Individual weights rarely have a meaning a person could name, which is the root of the difficulty in explaining a model’s behaviour that Module 9 examines. Developers also choose how much to disclose about their weights, and OpenAI’s technical report for GPT-4 in March 2023 withheld the model’s size, hardware, training compute and dataset construction altogether.