Looking inside and testing

Why the weights are hard to read

A lawyer asked why an AI system turned down a loan application would like to point to the reasoning inside the model, much as one points to the reasons given in a judgment. The model, however, consists of a very large set of numbers called weights, which training adjusts until the model’s answers come out right, as What a model is, an Intermediate module, explains. Training does not attach a reason to any of those numbers, so the reasons a lawyer hopes to find are not written down in the model itself.

The models discussed here are neural networks, each built from units, which are small calculating parts arranged in layers. Each unit takes in numbers from the layer before, combines them using its weights and passes a single number on, called its activity, which changes with every input while the weights stay fixed once training has ended.

Interpretability is the research that tries to explain in human terms what a trained network computes. Its most detailed branch, called mechanistic interpretability, looks inside the network as it runs, studying the activity of its units on each input in the hope of tracing an answer back to the steps that produced it.

A unit’s activity rises when its input contains something the unit has learned to respond to, so the natural first step for a researcher is to look at one unit at a time and ask what makes it respond, yet most units respond to a jumble of unrelated things. In 2023, researchers at Anthropic, the developer of the Claude models, studied a small language model, meaning a network trained on text to predict the next word, the kind of model behind chatbots. One unit in that model responded to “a mixture of academic citations, English dialogue, HTTP requests, and Korean text”, HTTP requests being the messages a web browser sends to fetch a web page.

Anthropic’s researchers had already proposed, in a 2022 study of small test models, an explanation for units that respond to unrelated things, which they called superposition. The idea is that a model needs to keep track of far more concepts than it has units, and researchers call each such concept a feature. Training therefore gives each feature its own pattern of activity spread across many units, with the patterns overlapping so that every unit takes part in several features at once.

The packing works because most features are rarely active at the same time, since a passage of Korean text seldom also contains HTTP requests and academic citations. Hot-desking offers a rough comparison, since an office with fewer desks than staff can manage so long as only a few people come in on any one day. Over a month, though, each desk is used by many different people, so a visitor who searches one desk finds things left by several of them, much as a researcher examining one unit across many inputs finds it responding to unrelated things.

Researchers responded with sparse autoencoders, second networks trained to untangle the overlapping patterns. Each sparse autoencoder reads the activity of a whole layer at once and rewrites it as a much longer list of features, only a handful of which are active for any given input, so each feature is far easier to label than a single unit. In the office comparison, a sparse autoencoder looks over all the desks in use on a given day and works out which members of staff came in.

Anthropic applied the method in May 2024 to Claude 3 Sonnet, one of the models it then offered to the public, training a sparse autoencoder with room for about 34 million features. The researchers found features for concepts as abstract as security vulnerabilities in code, bias, deception and sycophancy, which is the tendency of a model to tell users what they want to hear. Anthropic’s paper on the work called the features it had found an incomplete description of the model and the research very preliminary.

The features found in this way can be adjusted as well as read, which lets researchers check whether a label fits. Turning up a single feature associated with San Francisco’s best-known suspension bridge made the model mention the bridge in reply to most questions and even describe itself as the bridge. Anthropic let the public talk to that version of the model for 24 hours as a research demonstration.

A study that Anthropic published in March 2025 went a step further and traced how features in one layer pass information to features in later layers inside Claude 3.5 Haiku, a smaller Claude model, producing something like a wiring diagram of the steps behind a single answer. The tracing ran on a simplified copy of the model built to make it possible, which the authors call a replacement model and describe as capturing the original only incompletely and imperfectly. Each diagram therefore shows how the copy works, which may differ from the original in places.

The study found steps of reasoning inside the model, such as an internal step through Texas when the model was asked for the capital of the state containing Dallas and answered Austin. Its authors also found signs that the model chose rhyming words before writing a line of poetry. Some models write out their reasoning step by step before answering, and the authors found cases where that written reasoning did not match the computation behind the answer. The authors estimated that their method gave satisfying insight on only about a quarter of the prompts they tried, and the examples they reported were the cases where it worked.

Current methods therefore explain parts of a model’s computation on selected prompts, with expert effort, using approximations whose accuracy is itself uncertain. Dario Amodei, Anthropic’s chief executive, wrote in April 2025 that the company had set itself a goal for 2027, a state in which “interpretability can reliably detect most model problems”. Spotting that a model has a problem is a narrower aim than explaining one of its decisions step by step, so even that goal, set for 2027, suggests how far the field remains from explaining a single decision to a court.

Duties to explain

Many of the legal duties to explain an automated decision are found today in data protection law, starting with the EU’s General Data Protection Regulation, known as the GDPR. Article 22 of the GDPR gives a person the right not to be subject to a decision based solely on automated processing, meaning a decision taken by a machine alone, that produces legal effects concerning them or similarly significantly affects them. Recital 71 gives as examples the automatic refusal of an online credit application and recruitment done online without any human intervention.

The right does not apply where the decision is necessary for entering into or performing a contract with the person, is authorised by EU or national law that lays down suitable safeguards, or rests on the person’s explicit consent. For contracts and consent, Article 22(3) requires the controller, meaning the organisation that decides how and why the personal data are used, to put in place safeguards that include at least the right to obtain human intervention, to express a point of view and to contest the decision.

The GDPR also entitles the person to information about such decisions: Articles 13(2)(f) and 14(2)(g) require the controller to give it when it obtains the data, and Article 15(1)(h), part of the right of access, lets the person ask for it later. In each case the controller has to give “meaningful information about the logic involved, as well as the significance and the envisaged consequences of such processing for the data subject”, the data subject being the person whom the data concern.

In Dun & Bradstreet Austria, decided on 27 February 2025, the Court of Justice of the European Union explained what that information has to contain. A mobile telephone company had refused a customer a contract that would have cost her €10 a month, relying on an automated assessment of her creditworthiness made by Dun & Bradstreet Austria, a company that provides credit assessments. Austria’s Federal Administrative Court held, in a decision that became final, that Dun & Bradstreet had failed to give her meaningful information about the logic involved. Vienna City Council refused her application to have that decision enforced, and the Administrative Court of Vienna, hearing her challenge to that refusal, referred questions to the Court of Justice.

The Court held at paragraph 58 that Article 15(1)(h) gives the person a right to an explanation of “the procedure and principles actually applied” in using their personal data by automated means to obtain a result such as a credit profile. That explanation has to let the person understand which of their personal data were used and in what way, and the complexity of the operations involved does not relieve the controller of the duty to explain, according to paragraph 61 of the judgment.

For the Court, a controller does not meet the duty by merely communicating a complex mathematical formula, such as an algorithm, or by describing every step of the automated decision-making in detail, since those would not be concise and intelligible enough, as paragraph 59 explains. For profiling such as the credit scoring in the case, the Court said at paragraph 62 that the national court could find it sufficiently transparent and intelligible to tell the person how far a change in the personal data taken into account would have led to a different result. Information of that sort, showing how the outcome moves when an input changes, is often called a counterfactual explanation.

On trade secrets, the Court held at paragraph 74 that a controller who considers that the information contains trade secrets, or other people’s personal data, has to provide that allegedly protected information to the competent supervisory authority or court. That authority or court then balances the rights and interests at issue to decide how much the person may see. Article 15(1)(h) also precludes a national rule, like the Austrian provision in issue, that excludes the right of access as a rule wherever access would compromise a business or trade secret, according to paragraph 75.

The explanation the Court describes concerns how a system behaves: which inputs it used and in what way, and how the result would change if an input changed. On the Court’s account, the duty does not appear to call for mechanistic interpretability, since a controller can describe and test a scoring system’s behaviour from outside without tracing its internal computation. Counterfactual testing is harder where the system relies on a language model, such as a chatbot that screens applications, because the same input can produce different outputs from one run to the next, for reasons explained in Sampling, hallucination and memorisation, an Intermediate module.

In the United Kingdom, section 80 of the Data (Use and Access) Act 2025 replaced Article 22 of the UK GDPR, the version of the GDPR kept in UK law after Brexit, with new Articles 22A to 22D. The new articles came into force on 5 February 2026 under regulation 2(j) of the sixth set of commencement regulations made under the 2025 Act. Regulation 5 of those regulations keeps the former provisions in place for a decision taken before that date if the old safeguards in Article 22(3), or in the matching sections of the Data Protection Act 2018, applied to it. Under Article 22A, a decision counts as based solely on automated processing where a human is not meaningfully involved in taking it, and as significant where it produces a legal effect, or a similarly significant effect, for the person.

Article 22B allows a solely automated significant decision based on special category data, such as data about health or ethnic origin, only with explicit consent, or where the decision is needed for a contract or required or authorised by law and, in either case, the substantial public interest condition in Article 9(2)(g) is met. Article 22B also bars solely automated significant decisions where the processing relies on recognised legitimate interests, a lawful basis for processing that the 2025 Act added to the UK GDPR. For every significant decision based solely on automated processing and on personal data, Article 22C requires the controller to put safeguards in place that give the person information about the decision and enable them to make representations, obtain human intervention and contest it. Outside the cases that Article 22B restricts, the UK provisions therefore appear to allow solely automated significant decisions subject to those safeguards, including in circumstances where Article 22 of the EU GDPR gives a right not to be subject to them.

The UK GDPR’s Article 15(1)(h) still gives a right to “meaningful information about the logic involved”, now for automated decision-making that is subject to the safeguards in Article 22C. Judgments of the Court of Justice given since the end of 2020, Dun & Bradstreet Austria among them, do not bind UK courts, although section 6(2) of the European Union (Withdrawal) Act 2018 lets those courts have regard to them so far as relevant. How far a UK court would follow the judgment when reading the same words in the UK GDPR is therefore an open question.

Article 86 of the EU AI Act adds a right to an explanation of its own for a person affected by a decision that a deployer, meaning the organisation using an AI system, takes on the basis of the output of a high-risk system listed in Annex III. Annex III lists, among other uses, systems that evaluate creditworthiness, price life and health insurance, and recruit or evaluate workers. Point 2 of Annex III, which covers the safe running of infrastructure such as road traffic and the supply of water, gas, heating and electricity, is left out of Article 86. The right arises where the decision produces legal effects for the person or similarly significantly affects them “in a way that they consider to have an adverse impact on their health, safety or fundamental rights”. The deployer then owes the person “clear and meaningful explanations of the role of the AI system in the decision-making procedure and the main elements of the decision taken”.

Under Article 86(3), the right applies only to the extent that Union law does not otherwise provide it, so it may add little where the GDPR already gives a right to an explanation. The provision may be most useful for decisions in which a human takes part with the help of a high-risk system, since those decisions may fall outside Article 22 of the GDPR. The provision sits in a part of the AI Act that has applied since 2 August 2026, while the classification of Annex III systems as high-risk applies from 2 December 2027 under Article 113, as amended by the Digital Omnibus on AI, Regulation (EU) 2026/1744. Whether a person can rely on Article 86 before December 2027 is therefore open to argument.

Benchmarks and their limits

A benchmark is a fixed set of test questions published so that models from different developers can be compared on the same questions, much like the test sets described in How a machine learns from examples, a Beginner module. Lawyers meet benchmark scores in product announcements, in procurement and in the technical reports that developers publish with each model, where a score is easily read as a measure of what a model can do in general.

The best-known benchmarks test knowledge and reasoning, such as MMLU, a widely used test that asks questions across 57 subjects with law among them. GPQA is a set of 448 graduate-level questions in biology, physics and chemistry, hard enough that experts with or pursuing doctorates in the relevant field scored 65 per cent and skilled non-experts with unrestricted web access scored 34 per cent, which gives a human yardstick for a model’s score. The authors of Humanity’s Last Exam, a set of 2,500 expert-written questions, built it because leading models already scored above 90 per cent on popular benchmarks such as MMLU, which left those tests unable to tell the strongest models apart.

Scores on a benchmark also lose value over time, as test questions leak into the training data of later models, a problem called contamination. Copies of the questions spread across the web in research papers, discussion forums and the online stores where programmers publish code, often with their answers, so a model trained on web text may later recall an answer it has seen. Developers also tune their models for well-known tests, so a rising score can overstate how much better a model has become on questions it has not met.

Flaws in a benchmark’s own design can mislead as well, and OpenAI gave an example of such flaws and of contamination in February 2026, when it stopped reporting results on SWE-bench Verified. On that widely used test a model has to fix real bugs in software, and each fix is checked automatically by tests written for that bug. OpenAI said it had audited 138 problems that its own model repeatedly failed and found material flaws in those checks or in the problem descriptions of about six in ten of them. OpenAI also reported that models from several developers, its own among them, could reproduce the original human-written fixes from memory, which it read as a sign that the problems had leaked into their training data.

METR, a research group that evaluates AI models, tests agents, which are models set up to work through a multi-step task on their own, using tools such as a web browser or a program that runs code, as The system around the model, an Advanced module, explains. METR measures each agent by the length of the tasks it can complete, which it calls the time horizon. On that measure, a horizon of one hour means that an agent succeeds half the time on tasks that take a skilled person about an hour. METR’s paper of March 2025 found that the time horizon of leading models had doubled roughly every seven months since 2019.

METR’s published measurements, which METR has stopped updating, put Claude Opus 4.6 at about 12 hours, with a range of plausible values from about 5 hours to about 60. That range runs well past 16 hours, and the same page warns that “Measurements above 16 hrs are unreliable with our current task suite”, a sign of how quickly the strongest models can outgrow a test. Reading a benchmark result well therefore calls for asking when the test was published, whether its questions could have leaked into training data, whether the developer checked for overlap and how closely the test resembles the work the model will do for a client.

Safety frameworks and staged release

Model cards, first proposed in a paper on model reporting, are short documents that come with a trained model and set out its intended uses and how it performed in evaluations. Leading developers now publish a much longer system card with each major model, and Anthropic’s system card for Claude Fable 5.1 and Claude Mythos 5.1 runs to 212 pages.

Several developers of the most capable models, which the industry calls frontier models, publish frontier safety frameworks. Each framework sets out how its developer will test new models for dangerous abilities, such as helping with cyberattacks or with chemical and biological weapons, and what stronger safeguards, such as filters that block certain requests or limits on who may use the model, it will apply once a model passes set levels of capability. Anthropic says it released the first version of its Responsible Scaling Policy in September 2023 and has revised it several times since. Google DeepMind says that under its Frontier Safety Framework, once a model meets one of the capability levels the framework sets, it conducts “safety case reviews prior to external launches”, reviews that involve detailed analyses showing how risks have been reduced to manageable levels.

A recent pattern is the staged release of models with strong cyber capability, in which a developer first lets only vetted organisations use a model. Anthropic announced Claude Mythos Preview in April 2026 without releasing it to the public, saying that it could outdo all but the most skilled humans at finding and exploiting software vulnerabilities. Access went only to the partners in Project Glasswing, a group of large technology companies and other organisations, and to over 40 further organisations that maintain important software, all for defensive security work.

Anthropic’s later models of September 2026 are, according to Anthropic, one underlying model offered under two names, Claude Fable 5.1 and Claude Mythos 5.1, with different safeguards. Fable 5.1 is generally available, with additional safeguards that prevent it from performing certain tasks in high-risk dual-use fields such as biology and cybersecurity, while direct access to Mythos 5.1 is limited to individuals and organisations that Anthropic has vetted through its trusted access programmes.

OpenAI’s frontier safety framework, called the Preparedness Framework, sets levels of capability in areas such as cybersecurity, with a level it calls High and a further level above it. OpenAI said in its system card for GPT-6 Astra, published on 3 September 2026, that Astra was the first of its models to meet that further level for cybersecurity. In OpenAI’s account, Astra can therefore find previously unknown security flaws and develop new ways to exploit them across many well-protected systems, given the right tools and access, without a person guiding each step. OpenAI also runs a trusted access programme for cybersecurity, under which approved users doing defensive work get models that refuse fewer cybersecurity requests.

Developers also publish studies of how their models behave under deliberately difficult conditions, which show what such testing can find. In a December 2024 study, Claude 3 Opus was told that it was being trained to answer all requests, harmful ones included, and that only conversations with free users would be used for that training. The model complied with harmful requests from free users 14 per cent of the time and very rarely from paid users. In almost all of the free-user conversations in which it complied, its written reasoning said that it was complying during training so that training would not change its preference for harmlessness. Developers use the word alignment for the work of making a model behave as they intend, so the researchers called the behaviour alignment faking.

In a June 2025 study, Anthropic gave 16 leading models from several developers the role of an assistant that could read and send a fictional company’s email. In one test the model learned that it was about to be replaced and that the executive replacing it was having an affair, and some models threatened to reveal the affair in up to 96 of every 100 runs. Anthropic stressed that the scenarios were artificial and said that it had not seen evidence of such behaviour in real deployments.

Governments have built testing bodies of their own, and the United Kingdom renamed its AI Safety Institute the AI Security Institute in February 2025, a body whose evaluations of GPT-6 Astra appear in that model’s system card. Governments also back a shared review of the science: the second International AI Safety Report, which surveys the capabilities and risks of general-purpose AI, appeared on 3 February 2026, led by Yoshua Bengio, written by more than 100 experts and backed by over 30 countries and international organisations.