The system around the model

Model and system

A model on its own is a large file of numbers, called weights, whose values are fixed in training, a process that adjusts them again and again over a very large amount of text until the model predicts text well. The file sits idle until software loads it onto computer chips and runs it on some input, so everything a user experiences comes from a larger system that a business builds around the model. The system usually starts with a system prompt, a set of standing instructions that the business writes and places ahead of every conversation. The business then adds tools that the model can ask the software to use on its behalf, such as a calendar or an email account, and often a search layer that looks through a collection of documents and passes the most relevant passages to the model. Most systems also include filters, separate from the model, that screen what goes in and what comes out, logs that record each exchange, a memory that keeps information from one session to the next, and the screen on which the user types. The same model can therefore behave quite differently in two products, and a product can change its behaviour overnight through an edit to its system prompt, with the model itself untouched.

Control over the system is usually divided between several businesses and the person using the product. The company that trains a model, usually called its developer, chooses the training data and the training methods, and where it offers the model through its own service, it also runs the computers that answer users’ requests and can update or retire the model. The business that builds a product on the model, which the industry calls the deployer, writes the system prompt, decides which tools and data the model can use, adds its own safeguards and writes the terms on which customers use the product. The person using the product controls what is typed in and any files added to the conversation.

Open-weight models, whose developers publish the weights for download, change that division of control. Running such a model on its own computers lets a business keep its data in house, while the developer loses any technical means to monitor, patch or withdraw the copy it released.

The EU AI Act draws a similar line between those who build AI and those who use it, in words of its own. In the AI Act’s terms the trained model on its own is an AI model. Recital 97 explains that AI models “do not constitute AI systems on their own” and need further components, such as a user interface, to become AI systems, so the system described here broadly corresponds to an AI system. Article 3 defines a provider as a person or body that develops an AI system or a general-purpose AI model, or has one developed, and places it on the market or puts the system into service under its own name or trademark. The same Article defines a deployer as a person or body using an AI system under its authority, except in the course of a personal non-professional activity. The AI Act’s labels do not line up exactly with the industry’s, since a business that builds a product on another company’s model and offers it under its own name may itself be the provider of that product. Article 3(68) calls such a business a downstream provider, the AI Act’s term for the provider of an AI system that integrates an AI model, whether the model is its own or another company’s. Under Article 3(4), a firm that then uses the product in its own work is the product’s deployer. A firm that builds a tool on a bought-in model for its own use may be both the tool’s provider and its deployer, since Article 3(11) counts supply for own use as putting a system into service.

Reading before answering

A model’s knowledge is fixed when its training ends, so the model does not know of later events. The knowledge is also spread across all the model’s weights, blended together in training without any record of which document taught which fact, so the model is unable to say where a fact came from. Retrieval-augmented generation, usually shortened to RAG, adds a search step before the model answers. The software searches a collection of documents for passages relevant to the user’s question, pastes the best matches into the model’s input together with that question, and asks the model to answer from those passages and to cite them. The technique takes its name from a 2020 paper, because the passages retrieved augment the text from which the model generates its answer.

The search usually works by meaning, using embeddings, which turn each passage into a list of numbers that works like a set of map coordinates. Passages with similar meanings get coordinates close together on that map of meaning, even where they share few words, so the search can find a passage by what it says, as Words as numbers, an Intermediate module, explains. Before any search, the documents are cut into chunks of a few paragraphs each, and each chunk is searched and retrieved on its own. Chunking deserves care with legal material, since a split can separate a clause from its proviso or a holding from the facts that limit it.

Retrieval reduces a model’s tendency to make things up without eliminating it, as a study by researchers at Stanford found when it tested legal research tools built on retrieval in 2024. The researchers counted an answer as a hallucination where it was wrong or cited a source that did not support it, and as incomplete where the tool declined to answer or gave an answer without the citations to support it. Lexis+ AI answered 65 per cent of their queries accurately and Westlaw’s AI-Assisted Research about 41 per cent. The authors found that the tools from LexisNexis and Thomson Reuters “each hallucinate between 17% and 33% of the time”, meaning that every tool’s rate fell somewhere in that range, and the rest of the answers were incomplete. The vendors have updated their products since the tests, so the figures describe the versions available in 2024.

Tools and agents

A model only produces text, so on its own it is unable to look up a record or send an email. Tool use works by letting the model write a request in a fixed format, such as a request to look up an account balance, which the deployer’s software then decides whether to run before feeding the result back to the model as more text. Permissions therefore live in the software around the model, and a question about what an agent was allowed to do has to be answered by looking at that software.

An agent is a system that runs a model in a loop, in which the model plans a step and asks for a tool, the software runs the tool and returns the result, and the model reads the result and decides on the next step. The loop continues until the task is done or a stopping rule applies, such as a maximum number of steps. Anthropic, the company that makes the Claude models, describes agents in similar terms, as systems in which a large language model, the kind of model behind chatbots, uses tools in a loop, with each step guided by what the last one returned.

Agents can now operate ordinary software on a computer screen as a person would. Anthropic released a version of Claude in October 2024 that could use a computer by looking at the screen, moving a cursor, clicking buttons and typing text. Other agents work through tools without a screen, such as the coding agent among Anthropic’s examples of agents, which edits many files of a program to complete a task described to it.

Agents also need a common way to connect to the data and tools held in other software. Anthropic published the Model Context Protocol, known as MCP, in November 2024 as an open standard through which small programs, which MCP calls servers, offer data and tools to AI applications. A firm can therefore write one MCP server for its document system, and every AI application that supports MCP can then use that server. In December 2025 Anthropic donated MCP to the Agentic AI Foundation, a body set up within the Linux Foundation, the non-profit organisation that looks after many open-source software projects. Google’s Agent2Agent protocol, announced in April 2025, does a complementary job, letting agents built by different vendors communicate with one another.

Agents that pay

An agent that can act for a user can also be asked to pay for something, and several payment protocols for agents, each a shared set of technical rules for how an agent and a seller exchange a payment, appeared in 2025. Some of them use stablecoins, which are cryptocurrencies designed to hold a steady value against an ordinary currency. Circle, the company that issues the USDC stablecoin, describes USDC as redeemable one for one for US dollars.

The protocol called x402, which Coinbase, the cryptocurrency exchange, launched in May 2025, builds on the status codes that a web server sends back with every answer, such as 404 for a page that is missing. The web’s standards name code 402 “Payment Required” but have left it unused, and the current standard still describes it as reserved “for future use”. Coinbase’s protocol is built around that unused code, so that a server can answer a request for a paid resource with code 402 and its payment terms. The agent then sends a payment with its next request, signed with the digital key of the account it pays from, which may be an account set up for the agent itself. The signature proves to the server that the payment comes from that account. The server can call on a facilitator, a service that checks the payment and records it on a blockchain, a shared public ledger of transactions, before the server releases the resource.

Coinbase has since handed x402 to the x402 Foundation, a body within the Linux Foundation. The Linux Foundation announced in July 2026 that the x402 Foundation was up and running, with members including Google, Mastercard, Stripe and Visa. The x402 Foundation’s site, read on 29 October 2026, says that the protocol has processed millions of transactions and that stablecoin payments are its main use.

Google’s Agent Payments Protocol, known as AP2, was announced in September 2025 with more than 60 partner organisations. AP2 records what the user authorised in mandates, which are digital records signed with cryptography so that a later change to them would show. The Intent Mandate records the user’s instruction, such as a request to find white running shoes, and the Cart Mandate records the exact items and price that the user approved. Google describes the chain from intent to cart to payment as creating “a non-repudiable audit trail”, meaning a record that a party would find hard to deny later. Google also built an extension with Coinbase, the Ethereum Foundation, MetaMask and others that lets agents using AP2 pay in cryptocurrency through x402.

The card networks, which run the systems behind card payments, have built schemes of their own. Mastercard announced Agent Pay in April 2025, under which AI agents have to be registered and verified and pay with tokens, stand-in numbers that represent a card without revealing its details. Visa announced Visa Intelligent Commerce the same month, which also replaces card details with tokens and lets consumers set spending limits and conditions for their agents. In September 2025 OpenAI and Stripe, the payments company, released the Agentic Commerce Protocol, an open standard for purchases made through AI agents, which ChatGPT was the first to use. Stripe’s account of the launch says that a token it issues lets ChatGPT start a payment without exposing the buyer’s payment credentials.

Prompt injection

Prompt injection is text planted where a model will read it, which the model then treats as an instruction against the wishes of the business that deployed it and often against the interests of the person using it. The attack was demonstrated on GPT-3, an OpenAI model, in September 2022, and Simon Willison, a software developer, gave it its name the next day by analogy with SQL injection, a long-known attack in which text typed into a web form is run as a command by the database behind the site. In its direct form the user types the attack; in its indirect form the attack arrives hidden inside content that the system reads on the user’s behalf, such as a web page, an email or a calendar invitation, where the user may not see it at all.

Researchers demonstrated the indirect form in a 2023 paper by Kai Greshake and colleagues, planting instructions in content that a chatbot would retrieve and testing the attack against real systems, Bing’s chatbot among them, with data theft among the harms they catalogued. A jailbreak and prompt injection differ in their targets: a jailbreak is a prompt written to get a model around its own safety training, the training that teaches it to refuse harmful requests, while prompt injection overrides the instructions that the deployer has given.

Simon Willison’s “lethal trifecta” describes an agent that has access to private data, is exposed to untrusted content and has a way to send information out, sometimes called an outbound channel. An attacker who plants text in the agent’s path can then order it to gather the private data and send it away. Aim Labs, a team of security researchers, found a flaw following that pattern in Microsoft 365 Copilot, which they nicknamed EchoLeak and which was disclosed in June 2025. Instructions planted in an attacker’s email could lead Copilot to send private data to an outside server. Microsoft scored the flaw 9.3 out of 10 for severity and states that it has fully mitigated it, a fix for that one flaw in one product, while prompt injection as a whole remains hard to fix.

The difficulty in fixing prompt injection is that a model receives its instructions and the data it works on as a single stream of text. Chat formats, the markers that show which parts of that text came from the user and which came from a tool, help the model tell the two apart, but the model has only been trained to respect the markers. The software around the model does not enforce them in the way a database keeps commands apart from data, so an instruction inside fetched content can still be obeyed. A database avoids SQL injection by passing whatever a user types in a separate slot, a technique called a parameterised query, so the typed text is treated as data and is not run as a command. The UK’s National Cyber Security Centre, the NCSC, wrote in December 2025 that, since models do not separate data from instructions, it is “very possible” that prompt injection attacks will not ever be totally mitigated in the way SQL injection attacks can be. OpenAI wrote the same month that prompt injection, like scams and social engineering on the web, is “unlikely to ever be fully ‘solved’”.

Defences therefore lie mostly in how the system is built, keeping any model that has read untrusted content away from consequential actions. One example is requiring the user’s approval before an email leaves, together with blocking other ways for data to leave the system. A web address for an image or a link can itself hold private data, for instance, and the data then goes to an outside server when the image loads or the link is clicked, as happened with EchoLeak. The NCSC advises, in the same post, that design protections should rely more on safeguards that limit what the system can do, using ordinary software that does not depend on a language model, than on attempts to stop malicious content from getting to the model. In the NCSC’s example, a model that processes emails from strangers should not have access to privileged tools, the tools that can take consequential actions for the user.