The transformer

Where it came from

Every major chat assistant is built on a design called the transformer, so its strengths and weaknesses are shared by every chatbot a lawyer is likely to use. The transformer is a way of arranging a neural network, which is a program that learns its own settings, called weights, from examples. Researchers, most of them at Google, introduced it in 2017 in a paper titled “Attention Is All You Need”, as a better and faster way to build systems for machine translation, the task of turning text in one language into another.

Translation systems before the transformer used recurrent networks, a design that reads a sentence one word at a time, each step waiting for the one before to finish. The transformer looks at every word of a passage at once, which lets graphics processing units (GPUs), chips built to run many sums side by side, do the work in parallel and train a model in much less time.

Text entering a transformer is first cut into tokens, each a word or a piece of a word, and each token is given a vector, a list of numbers the model can calculate with. The vectors then pass up through a stack of identical blocks, and each block runs two layers in turn, an attention layer and then a feed-forward layer, where a layer is one round of calculation applied to every token’s vector. The model turns what comes out of the top of the stack into a prediction of the next token.

Attention

Attention is the step in which each token gathers information from itself and the tokens before it, which lets a model work out how a word depends on something said earlier. In “The tribunal dismissed the appeal because it was filed out of time”, a reader knows that “it” means the appeal, and attention is the part of a model that can make the same link. The same step is how a model can connect a defined term in a long contract to the clause that defines it, or a proviso to the clause it qualifies.

Attention links words by a kind of matching, in which each token asks a question and each earlier token, like the asking token itself, advertises what it holds. The asking token then takes in information from each of them, taking more from a token whose advertisement answers the question well and less from one whose advertisement answers it poorly. In the tribunal sentence, the question asked by “it” can match the advertisement of “appeal” closely, so “it” takes in information from “appeal” and later layers can treat the two words as linked.

Each token’s question, advertisement and information are three lists of numbers, called its query, key and value, which the model works out from the token’s vector using three grids of weights learned in training. The model compares the current token’s query with its own key and the key of each earlier token, which gives a relevance score for each one, and converts the scores into percentages that add up to 100. Each token then receives a blend of the compared tokens’ values in matching proportions.

The step that turns scores into percentages is called the softmax, and it keeps every share above zero, makes the shares add up to 100 per cent and gives a higher score a larger share. The softmax gives the top score a much larger share when the scores are far apart and similar shares when they are close together, so a word with one clear match focuses its attention there while a word with several plausible matches spreads it among them.

A single attention operation follows one pattern of matching, so each layer runs many of them side by side, called heads, each with its own grids of weights for queries, keys and values. One head might track the word just before, while another might link pronouns to the nouns they stand for. Large models run dozens of heads in every attention layer, 96 in the case of GPT-3, OpenAI’s 2020 model in the line that led to ChatGPT, and each head learns its own pattern in training.

A common picture of attention is a room of delegates, in which each delegate holds up a question card, wears a badge describing their expertise and keeps a briefing folder under one arm. Each delegate compares their question card with every badge in the room and takes a mix of the folders, weighted towards the badges that best answer the question. The picture is only a guide, since the cards and badges in a real model are lists of numbers produced by learned weights, without meanings a person assigned, and researchers have not yet named what many heads do. A chat model also limits each delegate to itself and the delegates before it, since the model is built to predict each word from the words that precede it.

Feed-forward layers and stored facts

The feed-forward layer is a small network of its own, which each token’s vector passes through separately after attention, and its name comes from the numbers flowing straight through it in one direction. Research suggests that much of what a model knows about the world, such as which city a landmark is in, is stored in the feed-forward layers, which therefore come up when a lawyer asks whether a fact, or a person’s data, can be found inside a model and removed.

Inside the feed-forward layer, a first stage widens the token’s list of numbers to about four times its length, and each new number works like a detector, its size showing how strongly one pattern is present in the token’s list. An activation function, a simple rule applied to each number, then keeps the detectors whose pattern is present and quiets the rest, and a second stage narrows the list back to its original length, adding what the active detectors contribute. The feed-forward layers hold roughly two thirds of the weights in a standard transformer, which led researchers to ask what so many weights do.

One study described each feed-forward layer as a memory with keys and values, using those words in a different sense from attention. Each detector acts as a key that recognises a pattern in the input, and its contribution in the second stage acts as a value that pushes the prediction towards the words likely to follow that pattern. Researchers went further in 2022 with a technique called Rank-One Model Editing, or ROME, which changes one fact by adjusting the weights of a single feed-forward layer. In one example, they found a feed-forward layer partway up the stack where a model appeared to hold the link between the Eiffel Tower and Paris, and edited that layer’s weights so that the model placed the tower in Rome.

The idea that each fact sits in one place is contested, since later work found that the layer where a fact appeared to sit was often not the best layer to edit, which would be odd if the fact lived in that layer alone. Knowledge appears to be spread across many weights that also serve other facts, so an edit made to change one association can disturb others. Repeated editing also wears a model down: a study that edited a model with one fact after another found that it gradually forgot earlier edits and lost some of its ability at other tasks, before failing abruptly. A fact in a model is therefore unlike a record in a database, which can be found and deleted. Lawyers meet the editing research when a person asks a business to correct or erase personal data that its model has learned, since the business may be unable to point to where the data sit or to show that an edit removed them.

Depth, order and the running total

Each token’s vector travels up through every block in the stack, and each block adds its output to the vector without overwriting what is already there. The running vector that every block reads from and adds to is called the residual stream, and it works like a running total. A fair picture is a file passed down a line of specialists, each adding notes without erasing earlier ones, although the notes are numbers added into the same list, so one note can overlap with another or cancel it out. The overlapping notes help explain why what a model knows is hard to trace to a single place inside it.

Word order changes meaning, since a sentence in which the landlord sues the tenant uses the same words as one in which the tenant sues the landlord. Attention on its own does not register word order, since the match between two tokens depends on what each one contains, wherever the two sit in the text, so each token’s place is marked separately. Chat models can work out some of the order even without those marks, probably because a token that sees only the tokens before it can gauge roughly how far into the text it sits from the number of tokens it can see.

The 2017 transformer added a fixed pattern of numbers to each token’s vector to mark its place in the text. Recent models whose weights are published, such as Meta’s Llama 3, use a method called rotary position embedding, or RoPE, which adjusts each query and key according to the token’s place in the text so that the match between two tokens reflects how far apart they are. The method inside a closed commercial chatbot, whose weights are kept private, is known only where its maker has disclosed it.

Predicting the next token

At the top of the stack, the model converts the final vector at the last position into a score for every token in its vocabulary, the fixed list of tokens it knows, and a softmax turns those scores into probabilities. The model then picks a token, adds it to the end of the text and runs the whole stack again to choose the next one, a loop called autoregressive generation because each token the model writes is fed back in as input. Every word of a chatbot’s answer, a case name or a citation included, is built from tokens chosen one at a time by that loop. How the model picks from the probabilities, and why a citation assembled token by token can be invented, is explained in Sampling, hallucination and memorisation, an Intermediate module.

The 2017 transformer had two halves, built for translation: an encoder that read the source sentence in full, and a decoder that wrote the translation one token at a time, drawing on the encoder’s reading of the source and on the part of the translation already written. Chat models keep only the decoder, which suits open-ended writing and became the basis of today’s chat assistants. Google’s BERT model kept only the encoder, reading a whole passage in both directions, which suits tasks such as sorting documents into categories or finding the answer to a question within a passage. A tool that sorts or searches a firm’s documents may therefore run on a different kind of transformer from the chatbot that drafts its letters.

The context window is the maximum number of tokens a model can take in at once, covering the prompt, any documents, the conversation so far and the answer. The window sets the limit on how much of a long agreement, or of a bundle of documents, a tool can read in one pass. Windows have grown quickly, from 2,048 tokens in GPT-3 in 2020 to around one million in leading commercial models in 2026, which is roughly 750,000 English words on the common estimate of three quarters of a word for each token.

Long context is expensive, since in the standard design every token is compared with itself and every earlier token. Doubling the length of a prompt gives twice as many tokens, each compared with about twice as many others, so the attention work needed to read it roughly quadruples. The software that runs a model also saves the keys and values already worked out for earlier tokens in a store called the key-value cache, or KV cache, so that they need not be worked out again for each new token. The KV cache grows with every token in the conversation, so a long conversation costs more memory as well as more work for each token written.

Models may not read every part of a long input equally well, since a 2023 study found that the models it tested used information best when it appeared at the beginning or end of a long input, and worse when it sat in the middle.