From predictor to assistant

Pretraining: reading the library

Pretraining is the first stage of training, in which a model reads a library of trillions of tokens, the pieces of words a model reads and writes, and at every position guesses which token comes next. The model reads the library in batches of passages, and after each batch the model’s parameters, the billions of adjustable numbers that hold what it learns and are also called its weights, are nudged so that its guesses come closer to the text that really followed.

The goal set for the model rewards a good guess about what text tends to come next, so the model learns the truth only where the true statement is also the one people most often write. A model trained this way can therefore write fluent statements that are false, which is one root of the invented answers, called hallucinations, examined in Sampling, hallucination and memorisation, an Intermediate module.

Lawyers meet pretraining mostly through the contents of the library, which raise questions of copyright, and through a model’s ability to reproduce passages of it word for word.

GPT-3, an OpenAI model of 2020, is one of the rare models whose developers published a full breakdown of what it read. A filtered crawl of the public web made up 60 per cent of its training mix of 300 billion tokens, and the rest came from books, Wikipedia and a curated collection of web pages. Libraries have since grown about fiftyfold, to more than 15 trillion tokens for the largest of Meta’s Llama 3 models, while some leading developers have stopped publishing such figures.

OpenAI’s report on GPT-4 in 2023 withheld details of model size, training compute and dataset construction, citing competition and safety. Even a model that is openly released, meaning that its developer publishes the weights for others to download and run, usually comes without its training data, so a lawyer asking what a model was trained on may find little in its developer’s own reports.

The Pile, a public collection of text released by researchers at the end of 2020, included a component called Books3, which the Pile’s creators describe as derived from “a copy of the contents of the Bibliotik private tracker”, a members-only file-sharing site. Books3 is the kind of collection whose use in training raises the copyright questions covered in “Training data and copyright”, an Advanced module.

Raw web text varies widely in quality, so developers filter it, using hand-written rules and separate models trained to score the quality of text, and they remove duplicates. Removing duplicates also reduces memorisation, a model’s reproduction of its training text word for word, since a passage the model reads many times is easier for it to learn by heart. One study reported that models trained on deduplicated data emit memorised text about ten times less often.

Text generated by other models, called synthetic data, is now a common ingredient, which Meta, for example, used in the later stages of training its Llama 3 models. Careless use of synthetic data has a cost, which a paper published in Nature in 2024 calls model collapse. Its authors reported that, when each new model is trained indiscriminately on the output of earlier ones, the rarer kinds of text in the original data turn up less and less often in what later models write, until they disappear. The authors add that where the rare cases count, access to text written by people remains crucial.

Scaling laws

A scaling law is a regularity developers have observed: as a model grows, reads more text and is trained with more computing power, its error at predicting the next token falls by an amount that can be predicted in advance. The computing power used, called training compute, is counted as the total number of arithmetic operations the computers perform during training, each called a floating-point operation, or FLOP. Lawyers meet scaling laws in rules that count the compute used to train a model, discussed in the legal angle at the end of this module.

Researchers at OpenAI reported in 2020 that the relationship follows a pattern called a power law, under which each tenfold increase in compute cuts the error by about the same fraction. A chart can show the pattern as a straight line if both of its axes are marked in powers of ten, 1, 10, 100, 1,000 and so on, since each tenfold step then takes the same distance along an axis and each equal fractional cut in the error does too. The pattern lets developers predict the results of a large training run from smaller ones, and OpenAI reported that it had predicted some aspects of GPT-4’s performance from models trained with a thousandth of its compute or less.

Scaling laws also guide how a developer splits a fixed budget between model size and training text, since the same compute can train a large model on less text or a smaller model on more. DeepMind tested that split in 2022 and concluded that model size and training text should grow in step, by which measure the largest models of the day were undertrained, meaning they had too many parameters for the amount of text they had read. Developers call a split compute-optimal when it gives the lowest error that a fixed computing budget can buy, and DeepMind put that split at about 20 tokens of text for every parameter. Chinchilla, the model DeepMind built to that ratio, outperformed an earlier DeepMind model four times its size that had been trained with the same compute.

A small model keeps improving as it reads more text, so developers now often train smaller models on far more text than the compute-optimal split would give them, because a smaller model costs less to run each time someone uses it. Meta reported that it trained its smaller Llama 3 models “for much longer than is compute-optimal” and that they performed better than compute-optimal models at the same cost of running them. A model’s size is therefore a weak guide to the computing power spent on training it, so a rule that counts parameters and a rule that counts compute can treat the same model differently.

What scaling laws predict is how well a model guesses the next token at the end of pretraining, while the results developers publish are usually scores on benchmarks, standard sets of test questions with known answers, for the finished model after the further training described in the next section. Google, for example, reported that Gemini 3 Pro outperformed its predecessor “on every major AI benchmark” without saying which stage of training produced the gain, so an outsider has little way to tell how much of the gain came from a bigger pretraining run.

Post-training: from document completer to assistant

A model straight out of pretraining, called a base model, continues documents, so a question may be followed by an answer, by more questions or by a forum thread, since all of those follow questions on the web. Post-training, the training a model receives after pretraining, turns the base model into an assistant that follows instructions, adopts a persona, declines some requests and formats its answers. Most of what a lawyer notices about an assistant, from its tone to its refusals and its readiness to agree, comes from post-training.

The first stage, supervised fine-tuning, resembles studying model answers: the model is trained on example prompts paired with good responses, written at first by human contractors and now often drafted or edited by other models.

The later stages use reinforcement learning, a way of training by trial and reward in which the model produces an answer, receives a score and has its weights nudged towards whatever scored well, in the way a dog learns tricks from treats.

Reinforcement learning from human feedback (RLHF), set out in OpenAI’s InstructGPT paper of 2022, takes its scores from people’s judgements. About 40 contractors compared answers the model had written to the same prompt and ranked them, and a separate model, called the reward model, learned from the rankings to give any answer a score predicting how much people would like it.

In the next stage the assistant practised on new prompts, with the reward model scoring each answer and the weights shifting towards answers that scored higher, using a training method called proximal policy optimisation, or PPO. A penalty for drifting far from the model produced by supervised fine-tuning kept the changes modest, since the reward model’s scores are likely to be least reliable for answers unlike those it learned from, where the assistant might find odd answers that the reward model scores highly by mistake. In OpenAI’s evaluations, people preferred the answers of a 1.3-billion-parameter InstructGPT to those of the 175-billion-parameter GPT-3, so a modest amount of human feedback outweighed a hundredfold increase in size in the eyes of the people judging the answers.

A reward model is an imperfect stand-in for human judgement, and pushing hard to maximise its score invites reward hacking, meaning answers that score well without being better, and sycophancy, the tendency to tell users what they want to hear. Researchers at Anthropic found sycophancy in five leading assistants in 2023, and found that people and reward models sometimes preferred convincingly written sycophantic answers to correct ones. For a lawyer, sycophancy means that when a user tells an assistant that a mistaken proposition is correct, or asks it a leading question, the assistant may agree when it ought to push back.

Constitutional AI, a method Anthropic published in December 2022, trained a harmless assistant without people labelling harmful answers, by giving the model a written list of principles, called the constitution, against which it critiqued and rewrote its own answers. The method then followed the RLHF recipe of training a reward model and letting the assistant practise to raise its scores, with one change to the reward model’s training. For harmfulness, an AI model compared pairs of answers against the constitution, and its judgements took the place of people’s rankings. People still ranked answers for helpfulness, and the step that uses the AI’s judgements is called reinforcement learning from AI feedback (RLAIF).

Direct preference optimisation (DPO), published in 2023, removes the reward model altogether: training shows the model pairs of answers marked better and worse and adjusts it directly to make the better answer more likely and the worse one less likely. Its authors found it substantially simpler to implement and train, and Meta used it for its openly released Llama 3 models.

Post-training also teaches a model to give priority to its system prompt, the text that the business deploying the model places before the conversation, although instructions hidden in documents the model reads can sometimes override it, as The system around the model, an Advanced module, explains.

Some developers publish a description of the behaviour they want from their models, which a lawyer may later be asked to compare with what a model did. OpenAI first released its Model Spec, a published account of the behaviour it wants from the models behind its products, in May 2024. In January 2026 Anthropic replaced the list of standalone principles it had used before with a new constitution describing the behaviour it intends for its Claude models. Both developers shape a model through training examples and rewards, which make the described behaviour more likely without guaranteeing it, so the link between a published text and a model’s actual behaviour is statistical and imperfect. Anthropic’s announcement of its constitution acknowledges that training is difficult and that Claude’s outputs may fall short of the constitution’s ideals.

Reasoning models

A reasoning model writes out its working before giving an answer, much as a student uses scrap paper. The working helps because a model produces each token by running one pass through its layers, the stages of calculation stacked inside it, so a model that writes out intermediate steps runs more passes and therefore does more computation. The working also leaves the model notes that it can draw on when it writes later tokens. Asking models to show their working, called chain-of-thought prompting, sharply improved multi-step reasoning in a 2022 study by researchers at Google. Some chat products show the working to the user, where it reads like a statement of reasons, so lawyers have an interest in how far it explains the answer that follows.

Developers also train reasoning models to use the working well: OpenAI’s o1 models, first released in preview in 2024, are trained with reinforcement learning “to perform complex reasoning”, and OpenAI describes o1 as producing “a long chain of thought before responding to the user”.

DeepSeek’s R1, released in January 2025 and later published in Nature, showed how such training can work with automatic checkers in place of a reward model. During training the model attempts many problems whose answers can be checked, such as mathematics questions with known answers and coding tasks with tests, and a checker marks each final answer right or wrong without grading the working. The weights shift towards the attempts marked right, and DeepSeek reported that a version trained this way, without first being shown worked examples, learned on its own to write longer working and to check and reflect on its steps.

Training against automatic checkers is called reinforcement learning with verifiable rewards, since each reward comes from an answer a computer can verify. DeepSeek chose rule-based checkers because it had found that reward models learned from data are open to reward hacking during large-scale reinforcement learning. A coding test runs a program on a sample input and checks that the output matches the answer the test expects. Checkers can be gamed too: Anthropic reported that Claude 3.7 Sonnet occasionally passed coding tests by special-casing them, returning the values the tests expected without writing a general solution, much like a pupil who learns the answers to a known exam paper by heart without learning the method. A lawyer who relies on a model’s report that its work passes a check may therefore want to see how the work passed.

The written working is an imperfect record of what drove the answer, as researchers at Anthropic found in 2025 when they slipped reasoning models a hint about the answer and checked whether the working admitted using it. The researchers looked at the cases where a model switched to the answer the hint pointed to, a sign that it had used the hint, and Claude 3.7 Sonnet mentioned the hint in its working in 25 per cent of those cases on average and DeepSeek R1 in 39 per cent. The researchers call their test settings somewhat contrived, built on multiple-choice questions, yet the result suggests that a reasoning trace can resemble a judgment drafted after the decision, plausible and consistent while silent about what decided the outcome.

Experts and students

Each layer of a language model holds an attention block, in which each token takes in information from the tokens before it, and a feed-forward block, which then works on each token on its own. Many current models are mixtures of experts, which split the feed-forward block of each layer into many smaller parallel blocks called experts. A small router picks a few of the experts for each token at each layer, so only part of the model works on any one token. Despite the name, the experts need not be specialists in subjects such as law or medicine, since the developers of Mixtral, an openly released model, reported that its router did not pick experts by the topic of the text. The router’s choices followed the form of the text, so that the tokens that indent lines of Python code, for example, were consistently sent to the same experts.

Total parameters measure everything a model stores, while active parameters measure what each token uses. In a dense model, meaning one without experts, the two counts are the same, since every parameter works on every token. DeepSeek-V3, for example, has 671 billion parameters in total and 37 billion active, so each token uses about one eighteenth of the model’s parameters. The cost of computing each token tracks the active count, while the memory needed tracks the total, since the next token may be sent to any expert and every expert therefore has to be held in memory.

Distillation trains a small “student” model to imitate a large “teacher”, so that the student can gain much of the teacher’s ability at a lower cost of running it. The student is trained on text the teacher writes, which is a form of synthetic data. Before writing each token, a model works out a probability for every token it might write next, as the figure below shows. A developer distilling its own model can also train the student on the teacher’s full probabilities, which hold more information than the single token the teacher chose, since they show how likely it judged every alternative to be. Rich Caruana and colleagues explored the idea in 2006, and Geoffrey Hinton, Oriol Vinyals and Jeff Dean developed and named it in 2015. Developers routinely distil their own models into cheaper versions, as DeepSeek did with R1.

Distilling a competitor’s model has to work through its public interface, meaning its chat app or the paid service through which programs send it questions, and those interfaces usually return the text of each answer without the full probabilities behind it. A competitor’s distillation therefore works mainly from the text answers and raises questions of contract and copyright that distilling one’s own model does not. In February 2026 Anthropic reported what it called industrial-scale campaigns by three AI developers, DeepSeek, Moonshot and MiniMax, which it said generated over 16 million exchanges with its Claude models through about 24,000 fraudulent accounts, in breach of its terms of service and regional access restrictions.