How a chatbot writes an answer

Text cut into pieces

The chatbots lawyers now use are built on a large language model, which is a neural network trained on a very large amount of text to predict what comes next. The network itself is a very large set of simple calculations, each controlled by settings that work like volume dials, and a large language model has billions of those dials. Training sets the dials: the network guesses the next word in passages of published text, and after each guess every dial is nudged a fraction in the direction that would have made the guess better. Both the network and its training are explained in more depth in Neural networks without the mathematics, a Beginner module.

The network works only with numbers, so before the model reads a sentence, software cuts the text into pieces called tokens and swaps each piece for its number in a fixed list. Common words become single tokens, while rarer words are cut into two or more pieces; punctuation marks and spaces are turned into tokens as well. The full list of pieces a model knows is called its vocabulary, and it runs to tens of thousands of entries or more.

A token is on average shorter than a word, so a text holds more tokens than words. As a rule of thumb, OpenAI’s documentation puts a token of English text at about four characters, or three quarters of a word, which makes about four tokens for every three words. A page of a judgment of around 500 words therefore comes to roughly 670 tokens. The limit on how much text a model can take in at once is measured in tokens, as is the price providers charge for each use, so a long bundle’s token count bears on how much of it a chatbot can read in one go.

The model sees only the numbers, which explains some odd failures. Asked how many letters a word contains, a model may get the count wrong, since the word arrived as one or two numbers and the letters inside it were not shown to the model. The way text becomes tokens is covered in detail in Words as numbers, an Intermediate module.

One piece at a time

The model takes the text so far and gives each piece in its vocabulary a likelihood of coming next, expressed as a percentage, with the percentages for all the pieces adding up to 100. The software around the model then picks one piece, adds it to the text and runs the model again on the longer text to choose the piece after that. Every answer a chatbot gives is written in this loop, one piece at a time, which is why the answer appears on screen a word or so at a time.

The simplest way for the software to pick is to take the likeliest piece every time, and some systems do just that. Chatbots often use a weighted draw, like a lottery in which each piece holds tickets in proportion to its likelihood, so a piece with a likelihood of 30 per cent wins about three draws in ten.

The same question asked a second time gets fresh draws, which can land on a different piece early in the answer. Every later piece is then predicted from text that includes the different piece, so the rest of the answer can take a different path and end up worded differently. A lawyer who has checked one answer has therefore checked that answer alone, since asking again can produce a different one.

A phone keyboard that suggests the next word works on a related idea with far less text behind it. A large language model has learned from a very large amount of published text, so its guesses take account of grammar, of facts that are often repeated, of the style of a legal opinion and of the shape of an argument. The lab below lets a reader play the software’s part, choosing each next word or letting the software draw one by its likelihood.

Why it sounds sure of itself

Each piece is chosen for being likely to come next, which is a different test from whether the sentence it completes is true. Fluent, confident prose in the register of a legal opinion is the kind of text the model has learned to expect, so the model writes in that register whether its statement is right or wrong. Checking a statement against a source is a separate step, and the model on its own does not take it. The name for a false statement written in that fluent, confident style is a hallucination.

The Divisional Court described the same link between prediction and error in R (Ayinde) v London Borough of Haringey, a judgment of June 2025 on lawyers who had cited cases that do not exist. At paragraph 6 it said that freely available generative AI tools “generate textual responses by predicting what words or phrases come next in a particular context, based on patterns identified from a vast quantity of training data”, and that their “coherent and plausible responses may turn out to be entirely incorrect”.

The way models are trained and tested may add to the confident tone, by rewarding a model for giving an answer where it could admit doubt. Developers test a model by putting sets of questions with known answers to it and counting how many it gets right. A later stage of training, after the model has learned from published text, works in a similar way, nudging the model’s dials towards answers that score well. Scoring that counts only right answers works like a multiple-choice exam that does not deduct marks for wrong answers, in which a student who guesses scores more on average than one who leaves a question blank. A 2025 paper by researchers at OpenAI and the Georgia Institute of Technology argues that language models produce false statements because “the training and evaluation procedures reward guessing over acknowledging uncertainty”. On the researchers’ argument, a model trained and tested that way learns that a confident answer scores better than an admission of doubt.

Instructions and documents added

A chatbot product adds text of its own to what the user types, before the model reads any of it. The business running the product writes a system prompt, which is a set of instructions placed ahead of the conversation. One example is a law firm that runs its own chatbot and tells it, in the system prompt, to answer only questions about employment law, in a formal tone, citing the firm’s own guidance. The user is not shown the system prompt, so a lawyer using the product may read an answer shaped by instructions the lawyer has not seen. The model gives the system prompt priority over the user’s own messages where the two conflict, because in the later stage of training, after it has learned from published text, answers that follow the system prompt score better.

Many products also search a collection of documents, such as a database of judgments or a firm’s own precedents, before the model answers the user’s question. The software finds the passages that seem most relevant to the question, pastes them into the text just before the question itself and asks the model to answer from them and to cite them. The technique is called retrieval-augmented generation, a name given to it in a 2020 paper, because the passages retrieved augment the text from which the model generates its answer. The model predicts each piece from all the text in front of it, so once that text includes a passage stating the answer, the likeliest continuation is one that draws on the passage. With the right passage in front of it, the model is therefore less likely to invent an answer.

Retrieval helps without removing the risk, since the search can miss the passage a lawyer most needs and the model can still misread or misquote the passages it was given. A study of legal research tools by researchers at Stanford, published in 2024, found that tools from LexisNexis and Thomson Reuters which use retrieval produced fewer hallucinations than a general-purpose chatbot. Each of those tools still hallucinated “between 17% and 33% of the time” on the questions the researchers set, with every tool’s overall rate falling somewhere in that band.

System prompts and retrieval are covered in more depth in The system around the model, an Advanced module. Models that write out their reasoning before they answer, and the reasons models invent authorities, are the subject of Does AI reason?, a Beginner module.