Does AI reason?

Writing out the steps

The chatbots lawyers use are built on a language model, a very large set of numbers that a computer uses to work out which words are likely to come next in a text. The numbers are set in training, when the model reads a vast body of published text, judgments sometimes among it, and is adjusted bit by bit until its guesses at the next word follow the patterns of that text. A sentence is therefore likely, to a model, when it fits the patterns the model drew from its training text, which is a different thing from being true.

The model writes its answer one piece at a time, each piece being a short word or part of a longer word. For every piece, the computer runs the model’s whole set of calculations once to work out how likely each possible next piece is. The computer then picks a piece, often by drawing at random in proportion to those likelihoods, so the piece it takes is a likely one without being the likeliest every time. The piece is added to the text, and the calculations run again on the longer text. Writing one piece at a time is covered in more depth in How a chatbot writes an answer, a Beginner module.

Each run of the calculations does a fixed amount of work, which may be too little for a problem of several stages when the model has to give the answer straight away. A model that writes out its working, the intermediate steps on the way to an answer, gets more of that work, since every step is more pieces of text and each piece brings another run of the calculations. Each step also stays on the page, where every later run reads it as part of the text, much as a student doing a long sum on scrap paper takes one stage at a time and reads back the figure written a moment before. Researchers at Google showed in a 2022 paper that asking models to write out a chain of reasoning, which they called chain-of-thought prompting, improved their results on arithmetic, everyday reasoning and logic puzzles.

Developers now train models to write out working without being asked, in a further stage of training that uses problems with answers a computer can mark. Examples are sums with known answers and requests to write a computer program that has to pass tests. A test, in programming, is a short check written in advance by whoever set the problem, which runs the program and confirms that it does what was asked. For each problem the model writes out working and then an answer, and the developers reward the attempts that end in the right answer. Rewarding an attempt means adjusting the model’s numbers so that the model becomes more likely to write the kind of working found in that attempt, and over many problems the model learns to write out working by itself. The Chinese developer DeepSeek reported in a January 2025 paper that a model trained in this way took up habits such as checking its own steps, without being shown examples of those habits.

Models built like this are called reasoning models, and many chatbots now show the user a summary of the working, sometimes under a heading such as “thinking”. The working on screen is still text the model writes, each piece chosen because it is likely to come next, so a step can read well and still be wrong. The calculations behind each piece also stay out of sight, which limits how far a lawyer can read the working as an account of how the model arrived at its answer.

Where it helps and where it fails

Writing out working helps most on problems that break into stages, each giving a result the next stage uses: a calculation in several parts, a timetable worked out from a set of deadlines, or a definition in a contract that refers to three other definitions. Developers of reasoning models report their largest gains in mathematics and programming, the fields where a computer could mark the answers during training. A chain of contract definitions has stages of the same sort, though the result there has to be checked by a lawyer who reads the contract.

Written working can still mislead, since a mistake in an early step runs through to the end, written as confidently as the correct steps around it. The working also adds computation without adding knowledge, so a model whose training text did not include a judgment can reason at length and still not know what the judgment decided.

The written steps may also leave out the thing that in fact drove the model’s answer. Researchers at Anthropic tested how often the working leaves out a hint, in a 2025 study that put the same questions to reasoning models twice, once plainly and once with a hint pointing to one answer, such as a remark that a professor favoured it. The researchers treated a switch to the hinted answer as a sign that the model had used the hint, and read the working behind each switch to see whether it mentioned the hint. In many of the combinations of model and type of hint that the researchers tested, though not in all of them, the working acknowledged the hint in fewer than one case in five.

A model’s working can resemble a judgment drafted after the decision was made, which may read smoothly and consistently while saying little about what decided the case. The working is therefore evidence of how the model presents its answer, and weaker evidence of how the model arrived at it.

Invented authorities

A case citation has a very regular shape: the parties’ names, a year, a court and a number, often with a proposition attached. The model behind a chatbot has seen a great many citations in its training text and has learned their shape thoroughly. A particular minor decision, by contrast, may have appeared in that text a handful of times or not at all, so the model has learned little about it as a specific fact.

Asked for an authority, the model writes text that is likely to come next by the patterns of its training text. Text likely to follow a request for an authority has the shape of a citation: a plausible pair of names, a believable neutral citation and a holding that supports the proposition asked about. Each part is likely, and the whole may describe a case that does not exist. A 2025 paper by researchers at OpenAI and the Georgia Institute of Technology explains why a fact seen rarely in training tends to come out wrong, using a person’s birthday as its example. In the researchers’ account, a birthday seen only once in the training text gives the model too little to go on in telling the true date from a plausible false one.

Reasoning does not cure invented citations, since a model can reason only from what it has learned, and careful steps built on an invented case are still built on an invented case. A request to say when it is unsure is answered in the same way, with likely text, so the model may still give an invented citation without a sign of doubt. In Ayinde, a June 2025 judgment on fake citations, the Divisional Court said at paragraph 87 that it did not consider the risks materially reduced by “asking the tool not to provide an answer if it is not sure”. The court said, in the same paragraph, that the safeguard is “to check any output by reference to an authoritative source”.

A worked example

On 6 June 2025 two judges of the High Court in England and Wales, Dame Victoria Sharp P and Johnson J, sitting together as the Divisional Court, gave judgment in two cases that other judges had referred to them. Both cases were heard under the court’s jurisdiction to regulate its own procedures and enforce the duties lawyers owe to the court, and the judgment is reported as R (Ayinde) v London Borough of Haringey, heard with Al-Haroun v Qatar National Bank, [2025] EWHC 1383 (Admin).

In Ayinde, grounds for judicial review against Haringey Council, drafted and signed by a pupil barrister, cited five cases that do not exist. One of the five was given as R (on the application of El Gendi) v Camden LBC [2020] EWHC 2435 (Admin). The court found, at paragraph 36, that a case of that name does not exist, and that the neutral citation belongs to a different case about a charity’s liability to pay business rates. The grounds also misstated section 188(3) of the Housing Act 1996 as imposing a duty to secure accommodation pending a review, though the section says an authority “may” do so.

The barrister denied using AI, and the court did not decide whether she had. The Divisional Court said, at paragraph 68, that there seemed to be two possible explanations: either the fake citations were included deliberately, or generative AI was used and the denial was untruthful. In the same paragraph the court said that either explanation would be a contempt. The court found the threshold for contempt proceedings met, decided not to start them, for reasons it set out at paragraph 69, and referred the barrister to her regulator. The solicitor in charge of the law centre that instructed her was also referred to the Solicitors Regulation Authority, to consider the steps he took once the council’s solicitor had pointed out the fake citations.

In Al-Haroun, a claim against Qatar National Bank, correspondence and witness statements put before the High Court cited 45 authorities. Paragraph 74 records that 18 of the cases did not exist, and that many of the others did not contain the quotations attributed to them or did not support the propositions for which they were cited. The client, Mr Al-Haroun, said the citations came from publicly available AI tools, legal search engines and online sources, and his solicitor said he had relied on that research without checking it. One of the fake authorities cited to Dias J, the judge dealing with the claim, was a decision attributed to Dias J herself. The Divisional Court found the threshold for contempt not met and referred the solicitor to the Solicitors Regulation Authority.