Sampling, hallucination and memorisation

Choosing the next token

A chatbot such as ChatGPT is built around a language model, a program that has learned to predict the next piece of text by studying a very large collection of writing, called its training data, in a process called training. The language model works in tokens, a token being a word or a piece of a word, and its vocabulary is the fixed list of tokens it knows, usually tens of thousands of them or more.

Given a prompt, meaning the text the user types in, the model writes its answer one token at a time, and at each step it gives every token in its vocabulary a probability of coming next. The rest of the chatbot, which this module calls the software around the model, then picks one token from those probabilities, and the way it picks explains why asking the same thing twice can produce two different answers.

Greedy decoding, the simplest way of picking, takes the most probable token at every step, so in theory the same prompt produces the same answer on every run, a theory that tiny rounding errors inside the computer can upset. Sampling draws a token at random in proportion to the probabilities, like a lottery in which each token holds tickets in proportion to its percentage, so a token with a 25 per cent probability is picked about one time in four. Answers produced by sampling vary from one run to the next, a property called nondeterminism, so a lawyer who retypes the prompt behind a disputed answer may get back different text.

The probabilities come from raw numbers called scores: the model gives every token a score, and a formula called the softmax turns the scores into percentages that add up to 100. The softmax raises a fixed number, about 2.7, to the power of each score and divides the 100 per cent in proportion to the results. Only the gaps between scores therefore count: a token whose score is one point higher than another’s gets about 2.7 times the other token’s share, whatever the two scores are.

Temperature is a setting that controls how adventurous the sampling is, by rescaling the scores before the softmax: the software divides each score by the temperature, so a temperature of 1 leaves the percentages as the model computed them. A temperature below 1 enlarges every score and so widens the gaps between them, which lengthens the lead of the most likely token; a temperature above 1 shrinks the gaps, so less likely tokens come up more often. The provider of a chatbot or the developer of a tool usually chooses the temperature, and the person typing a question often does not see it.

With illustrative scores of 2.0, 1.0 and 0.1 for the words “court”, “tribunal” and “judge”, a temperature of 1 gives probabilities of 65.9, 24.2 and 9.9 per cent. Lowering the temperature to 0.5 doubles every score, to 4.0, 2.0 and 0.2, which doubles the gaps and sharpens the probabilities to 86.4, 11.7 and 1.9 per cent, so “court” wins almost nine draws in ten. Raising the temperature to 2 halves the scores and the gaps, flattening the probabilities to 50.2, 30.4 and 19.4 per cent, so “judge” comes up about one draw in five.

Top-p sampling, also called nucleus sampling, is a second setting, which trims the list of candidates before the draw. Under top-p, the software ranks the tokens from most to least likely and keeps adding them until their running total first equals or passes a chosen share, such as 90 per cent. The token that takes the total past the share stays in, and every token after it is left out of the draw. With the share set at 90 per cent and a temperature of 1, “court” and “tribunal” together make up 90.1 per cent, so “judge” is left out. The software then scales up the percentages of the tokens kept so that they add up to 100 again, as the chances in any draw have to, giving 73.1 per cent for “court” and 26.9 per cent for “tribunal”.

Setting the temperature to zero amounts to greedy decoding, since dividing by zero is impossible and the software treats a temperature of zero as an instruction to pick the most likely token at every step. In theory the answer is then the same every time, yet researchers at Thinking Machines Lab, an AI company, showed in 2025 that answers can still vary at a temperature of zero because of the way computers do arithmetic.

Computers store numbers with a limited number of digits and round after every addition, so the order in which numbers are added can change the total. With rounding to whole numbers after each step, adding 0.6 to 0.6 gives 1.2, which rounds to 1, and adding 10 then gives 11. Starting from 10, adding the first 0.6 gives 10.6, which rounds to 11, and adding the second 0.6 gives 11.6, which rounds to 12.

Working out each score means adding up a great many numbers, so the order in which they are added can shift the score very slightly. Models run on graphics chips, processors first built for video games and now used for AI because they do many calculations at once, and a server handles requests from many users together, in batches. The software on those chips adds numbers in a different order depending on the size of the batch, which depends on how busy the server is, so other people’s traffic can change the arithmetic behind an answer.

The resulting difference in a score is tiny, yet it can tip the choice between two close tokens, and since each token is chosen in light of the ones already written, one changed word can send the rest of the answer down a different path. In one test the researchers sent the same prompt 1,000 times, at a temperature of zero, to a model running on their own equipment, and received 80 different versions of the answer. Running the model themselves ruled out a provider changing the model between requests. The 1,000 requests themselves supplied the traffic, since the server grouped them into batches whose size varied as they arrived, so the variation came from the arithmetic alone. A single transcript of a model’s output may therefore be impossible to reproduce exactly, even by the person who produced it, which bears on its use as evidence.

Why models make things up

A hallucination is a fluent, confident statement by a model that is false, such as a wrong birthday given for a real person. For lawyers, the most familiar example is an invented case citation, which is a professional risk for the lawyer who relies on it.

A paper on why language models hallucinate, published in 2025 by researchers at OpenAI and Georgia Tech, argues that the way models are trained and tested rewards guessing over admitting uncertainty. The paper adds that arbitrary facts, meaning facts that a model could not work out from other facts, give it too little to learn from when they appear only rarely in training.

The paper’s example of an arbitrary fact is a private person’s birthday, since knowing other facts about the person does not reveal the date. Mentioned many times in the training data, a birthday forms a pattern that the model can learn and repeat. Mentioned only once, a birthday gives the model a single example, too little to learn from, so when asked for it the model tends to produce a date that merely looks right and is likely to be false.

A base model, meaning one trained only to continue text, has not been taught to decline a question, so it answers with its best guess even when it has little to go on. The paper therefore predicts that if 20 per cent of birthday facts appear exactly once in the training data, a base model will give a false answer to at least 20 per cent of birthday questions. Its guesses at the once-seen dates are rarely right, and any mistakes on better-known birthdays come on top.

Most benchmarks, the standard tests used to compare models, mark each answer right or wrong and give zero credit for an admission of uncertainty, so a model that guesses scores better on average, as a student does in a multiple-choice exam without negative marking. On the paper’s account, developers tune their models to do well on benchmarks, so the habit of guessing is learned in training as well as rewarded in testing.

OpenAI illustrated the incentive with two of its own models on a factual quiz called SimpleQA. The older model answered 24 per cent of the quiz questions correctly and 75 per cent wrongly, while the newer one answered 22 per cent correctly, 26 per cent wrongly and declined the rest. Judged on accuracy alone, the older model looks slightly better, despite giving 75 wrong answers in every 100 against the newer model’s 26.

The percentages a model computes measure how likely each token is to come next, which is a different thing from whether a statement is true. An invented citation can therefore look likely word by word, since it follows the usual pattern of citations closely, with a court, a year and a page number in the expected places. Even so, the percentages hold a clue, since a model that has learned a fact tends to give the right answer a high probability every time it is asked, while a model that is guessing spreads its probability across many answers. Research has found that models can use signals of this sort to estimate, imperfectly, whether they know an answer: asked to rate their own answers, they tend to give the correct ones higher marks. Models still often guess when unsure, because training and testing reward a guess, so developers have to train them deliberately to decline.

Retrieval, in which the software around the model looks up source documents and gives them to the model along with the user’s question, reduces invention without eliminating it, as The system around the model, an Advanced module, shows with legal research tools. A citation from a tool built on retrieval is therefore still worth checking against its source.

Memorisation

A model’s weights, also called its parameters, are the billions of numbers that training adjusts, and they are saved together in a file known as a weight file. A model’s size is usually given as the number of its weights, so “70B” in a model’s name means 70 billion of them. The training texts are not kept alongside the weights, so arguments about whether a model holds a copy of a work come down to what the weights themselves contain.

Memorisation means that a model can reproduce a passage from its training data, word for word or nearly so, when prompted. Exact and near copies make up the memorised end of a range whose other end is learned facts and a learned style, which let the model write about a subject or in an author’s manner without reproducing any particular passage. Lawyers meet memorisation in copyright claims that a model is or contains a copy of the works it learned from, and in data protection questions about whether personal data can be drawn back out of a model.

Researchers commonly test for memorisation by giving a model the opening of a passage from its training data, often 50 tokens long, and checking whether it completes the next 50 tokens word for word. Researchers call a passage completed in this way an instance of discoverable memorisation, a label unrelated to disclosure in litigation, and the test needs access to the training data, since it starts from the true opening of a training passage. Extractable memorisation is the stronger finding, and the one closer to the risks lawyers deal with, since it shows that an outsider without the training data can make the model produce the passage. Checking a suspected match still needs a copy of the original from somewhere, so outsiders compare the output with public text, such as pages found on the web, and some studies have had the developer confirm the matches.

The first large demonstration of extraction came in 2020, with GPT-2, an OpenAI language model. Researchers generated large amounts of text from the model, checked the most promising outputs by hand and, with OpenAI’s help, confirmed 604 memorised training examples, including names, phone numbers, email addresses and code. Follow-up work found that memorisation grows with the size of the model, with how often a passage is duplicated in the training data and with the length of the opening used as a prompt. Duplication helps explain how one model can reproduce a stranger’s phone number word for word yet invent a stranger’s birthday, since the number may have appeared many times in its training data and the birthday only once.

Chatbots such as ChatGPT receive extra training, called safety training, that teaches them to behave as helpful assistants and to refuse unwanted requests, which raised the question whether that training removes memorised text or only hides it. In 2023 a team showed that asking ChatGPT to repeat the word “poem” forever eventually made it break out of its chat behaviour and emit memorised training text, at about 150 times the rate at which it does so when behaving normally. The researchers spent about 200 US dollars on queries and recovered more than 10,000 memorised examples, confirmed by matching them against a large collection of text gathered from the public web. The team concluded that the safety training given to chat models had hidden the memorisation without removing it.

Open-weight models, whose developers publish the weights for download, let researchers run the model themselves and read the probability it gives each token of a passage, which shows how likely the model is to produce that passage word for word. One 2025 study of such models, by A. Feder Cooper, Mark Lemley and colleagues, found that, with the extraction method they used, most models do not memorise most books. Meta’s Llama 3.1 70B was an exception for some titles: it had memorised the first Harry Potter book so thoroughly that the researchers could extract the whole book almost word for word, starting from its opening words.

Production chatbots, the versions that companies offer to the public, can also give back long stretches of a book, as a 2026 study of four of them found with the same Harry Potter book. One chatbot gave back 95.8 per cent of the book nearly word for word after a jailbreak, a prompt designed to get around safety training. Two others gave back more than 70 per cent without a jailbreak, while the fourth produced 4 per cent before refusing. The authors cautioned against reading the results as a comparison of the models’ overall risk of giving back memorised text, since they tested each chatbot under different conditions.

Memorisation concentrates where text is long, distinctive and heavily duplicated, which describes famous novels that appear across the web in copies, quotations and fan sites. Memorised text is also spread across many weights in a form that is hard to read off by inspecting the numbers themselves, so researchers find it by running the model on prompts and watching what comes out. A failed extraction attempt therefore does not prove that a passage is absent from the weights, since an extraction attack gives only a lower bound on what a model has memorised, a minimum that a better prompt might exceed.