Open weights, images and provenance

Customising a model

Fine-tuning means taking a model that another developer has already trained and training it further on a smaller collection of examples chosen for one task, such as a firm’s precedents or a bank’s letters to its customers. Training sets a model’s weights, the numbers it learns, by starting from random values and nudging each one a little at a time, example after example. Fine-tuning continues that nudging from where the original training stopped, so it moves the existing weights towards the new task without going back to random values. A business can fine-tune a model in this way on its own computers when its developer has published the weights for download, which makes it an open-weight model, a kind of release that the next section explains. The result is a new file of weights, which its maker can keep, sell or give away, subject to whatever the licence of the original model says.

Models are trained and run on graphics chips, which were first designed to draw images on a screen and suit models because they can do thousands of simple calculations at once. Retraining every weight of a large model takes a great deal of their computing power and memory, so researchers have looked for cheaper ways to fine-tune. One such method, LoRA, leaves the original weights untouched and trains a much smaller set of extra numbers, which are added to the original weights when the model runs, so the model behaves as if its weights had been changed. For GPT-3, a model with 175 billion weights, the LoRA paper reports that the method cut the number of weights being trained by a factor of 10,000 and the memory needed on the chips by a factor of three, compared with fine-tuning every weight.

A 2023 method called QLoRA combined LoRA with quantisation, which here stores each of the original weights in four bits where sixteen are usual, so the model takes up about a quarter of the memory. Its authors fine-tuned a model with 65 billion weights on a single graphics chip with 48 gigabytes of memory, and their best model needed about 24 hours of fine-tuning on that one chip. Fine-tuning a large model had therefore become possible on one chip in about a day.

That ease lets whoever holds a model’s weights train away the safety behaviour its developer added, such as the model’s habit of refusing harmful requests. Before releasing its open-weight gpt-oss models in August 2025, OpenAI tested that risk itself in what it called malicious fine-tuning, retraining the models to stop refusing unsafe requests and to be as capable as possible in biology and cybersecurity. OpenAI chose those two fields because misuse there could cause severe harm, such as help in creating biological threats or in breaking into computer systems. OpenAI reported that even the retrained models fell short of OpenAI o3, one of its own models. OpenAI had already rated o3 below the level of capability in those fields that its safety framework treats as high, so on that comparison the retrained models stayed below that level too. OpenAI said that the results contributed to its decision to release the open-weight models. A developer that publishes its weights gives up technical control over every copy, since any holder can retrain a copy as OpenAI’s test did; the licence and the law are then its main means of limiting what recipients do with them.

Open weights and licences

A closed model is offered only as a service: the developer runs it on its own computers, customers send it requests over the internet, and the weights stay with the developer. With an open-weight model, the developer publishes the weights, so a user can download them and run the model on its own computers. Publishing the weights is a separate step from publishing the training data or the code used for training, and a developer can take one step without the others.

Licences for open weights vary widely, and Meta’s Llama 4 Community License shows how much one of them can say. Section 1 grants a royalty-free licence to use, copy, modify and distribute the model, while section 2 withholds those rights from a licensee whose products or services, together with those of its affiliates, had more than 700 million monthly active users in the calendar month before Llama 4 was released. A business above that line therefore has to ask Meta for a licence, which Meta may grant or refuse as it chooses, and the business may not exercise the community licence’s rights until Meta grants them.

Section 1 also sets rules on attribution and naming: a business that distributes the model, or a product or service containing it, has to supply a copy of the licence and display “Built with Llama” prominently, for example on its website or in its product documentation. Any model that a business creates, trains or fine-tunes with Llama 4 or its outputs, and then distributes or makes available, has to have “Llama” at the start of its name.

Meta’s acceptable use policy for Llama 4 forms part of the licence and lists uses the licensee agrees to avoid, among them using the model to “intentionally circumvent or remove usage restrictions or other safety measures”. The policy also contains a regional carve-out for the multimodal Llama 4 models, those that work with images as well as text, withholding the rights granted in section 1(a) from individuals domiciled in the European Union and from companies with their principal place of business there. The carve-out does not apply to end users of a product or service that incorporates those models.

Other developers have chosen the Apache License 2.0, a permissive licence written for software. Under clause 2 each contributor grants a perpetual, worldwide, royalty-free and irrevocable licence to reproduce, adapt and distribute the work, and the conditions in clause 4 deal mainly with passing on a copy of the licence, marking changed files and keeping the original notices. Mistral AI, the Qwen team, OpenAI and Google have all released models under the Apache licence, such as Mistral Small 3, Qwen3, gpt-oss and Google’s Gemma 4.

Some licences go further and require each recipient to pass the developer’s restrictions on to the next, and Google’s Gemma Terms of Use, which cover the Gemma models before Gemma 4, are one example. Under section 3.1 a business that distributes Gemma, or a model derived from it, has to include the use restrictions of section 3.2 as an enforceable provision in any agreement governing its use or distribution, and has to tell later users that those restrictions apply. The terms count offering the model as an online service as distribution, and they define a derived model widely enough to include a model trained to imitate Gemma on text that Gemma generated, a technique called distillation. Stable Diffusion version 1.4, an image generator described below, was released under the CreativeML OpenRAIL-M licence, one of a family of open responsible AI licences. That licence also requires each recipient to pass its restrictions on, in its case the restricted uses listed in its Attachment A, among them fully automated decision-making that adversely affects a person’s legal rights.

Whether a licence with use restrictions or user thresholds counts as open source is disputed. The Open Source Initiative, which maintains the Open Source Definition used to approve software licences, said in July 2023 that the licence for Llama 2, an earlier Meta model, is not open source, because the definition forbids discrimination against persons, groups or fields of endeavour. The Initiative pointed to that licence’s limit on commercial use by some users and to its acceptable use policy, which restricts use of the model for certain purposes.

The Initiative’s Open Source AI Definition, released as version 1.0 in October 2024, asks a developer that calls its AI system open source to make available the code used to train and run it and the model’s parameters, such as its weights. The definition also asks for information about the training data detailed enough for a skilled person to build a substantially equivalent system. That information may describe data that the developer is unable to share, so the definition does not require the raw training data to be published.

The open-source label also has consequences under the EU AI Act, which sets duties for the providers of general-purpose AI models, meaning models that display significant generality, can perform a wide range of distinct tasks and can be built into many other systems, as Article 3(63) defines them. The AI Act uses the word provider for the business that develops such a model, or has it developed, and places it on the market under its own name, meaning that it first makes the model available in the EU, whether for payment or free of charge. Under Article 53(1) a provider has to keep technical documentation for the AI Office, the part of the European Commission that supervises these models, and give documentation to the businesses that build the model into their own systems. The same paragraph requires the provider to adopt a policy to comply with EU copyright law and to publish a summary of the content used to train the model.

Article 53(2) lifts the two documentation duties for providers of models released under a free and open-source licence that allows access, use, modification and distribution, with the weights and the information on the model’s design and use made public. The exemption does not apply to a model with systemic risk, the AI Act’s category for models with high-impact capabilities. The copyright policy and the training summary remain due, as recital 104 explains. In its guidelines on general-purpose AI models, the Commission lists terms that would disqualify a licence, among them “usage restrictions triggered by user scale thresholds”, such as a demand for further licensing once monthly active users pass a set number. On the Commission’s reading, a licence with a user threshold such as Llama 4’s would therefore appear to fall outside the exemption, although the guidelines say that they are not binding and that only the Court of Justice can give an authoritative interpretation of the AI Act.

Images, video and voices

Most image generators are diffusion models, which learn to build a picture out of random speckle by removing the speckle step by step. In training, the model is shown real images with increasing amounts of that speckle, called noise, added to them, and it learns to predict the noise so that it can take it away again. To make a new image, the generator starts from pure noise and removes a little at each step, steered by the words of the prompt, until a picture emerges that matches the description.

The idea was introduced in a 2015 paper, and a 2020 paper showed that diffusion could produce high-quality images. Latent diffusion, a method published in 2021, ran the noise-removal steps on a compressed version of each image, which cut the computing cost, and it became the basis of Stable Diffusion, an image generator whose weights were published for download.

Stable Diffusion’s first versions were trained, according to its model card, the description its developers published with the model, on subsets of LAION-5B, a collection of 5.85 billion pairs of images and captions gathered from the web. LAION, the German association behind the collection, distributes it as a table of links to images hosted elsewhere, together with their captions, though it downloaded images to check them against their captions while building the table. One photographer, whose picture LAION had downloaded in that way from a photo agency’s website, sued, and in Kneschke v LAION the Hanseatic Higher Regional Court in Hamburg held in December 2025 that the German exceptions for text and data mining, the automated analysis of works to find patterns, permitted the download.

The Hamburg court’s press release explains that the photo agency’s reservation of rights, the statement on its website reserving the photographs against text and data mining, was not in the machine-readable form that German law requires, and that the download was also justified as scientific research. The court allowed a further appeal to the Federal Court of Justice, and its press release states that the judgment is not final; the questions it raises are covered in “Training data and copyright”, an Advanced module.

Cloning a voice needs very little recorded speech, as OpenAI reported in March 2024: its Voice Engine model, which it had let a small group of partners test, could generate natural-sounding speech closely resembling a speaker from text and a single 15-second recording of that speaker. OpenAI said that it had chosen to show the technology to the public without releasing it widely.

Deepfakes, which the EU AI Act defines in Article 3(60) as AI-generated or manipulated image, audio or video content that resembles real people, objects, places or events and would falsely appear authentic, have already been used for fraud. Arup, a British engineering firm, confirmed in 2024 that fake voices and images were used in a fraud against it in Hong Kong. According to the Hong Kong police, as CNN reported, an employee there paid out about 200 million Hong Kong dollars, some 25 million US dollars, after a video call with people he believed were the chief financial officer and other colleagues.

Marking what machines made

Two families of technique aim to let people tell content made by a machine from content made by a person. Provenance records describe where a file came from, and Content Credentials, a standard published by the Coalition for Content Provenance and Authenticity, known as C2PA, are one example. Content Credentials attach to a file a record of its origin and its edits, including any use of AI, and store the record as metadata, the information about a file kept alongside its contents. The software or device that makes the record signs it digitally, a step that works like a tamper-evident seal over the file and the record together, so a later change to either breaks the seal and shows that something has been altered.

The C2PA’s explainer accepts that the record can be removed from the file. The standard therefore lets a copy of the record be kept in online storage and found again from the content itself, through an invisible watermark of the kind described below or through a fingerprint, a short code computed from what the content looks or sounds like and matched against a database. The explainer also says that Content Credentials do not judge whether the provenance information is true, only whether the record follows the standard’s format, is free from tampering and was signed by a party on a list of trusted signers.

A watermark hides a signal in the content itself, so the signal stays in the content when metadata such as a Content Credentials record is stripped away. Google DeepMind’s SynthID-Text, described in a paper in Nature in October 2024 and released as open-source code, works on text. A language model writes by choosing each next word from several likely candidates, and SynthID-Text changes only the way that choice is made, nudging the choices into a statistical pattern that a detector can recognise.

The paper on SynthID-Text accepts that edits to the text, such as paraphrasing by another language model, weaken the watermark, and that a watermark exists only where the generator chose to apply one. The authors add that a watermark is hard to enforce on open models that run on many people’s own computers; the watermark is applied by the software that picks each word, and whoever runs the model controls that software. A text without a watermark therefore proves little about who or what wrote it.

The EU AI Act now requires providers to mark what their systems generate. The duty concerns AI systems, the working products built around a model, since in the words of recital 97 a model requires further components, “such as for example a user interface”, to become an AI system. Article 50(2) obliges the providers of AI systems that generate synthetic audio, images, video or text, including general-purpose systems such as chatbots, to ensure that the outputs are “marked in a machine-readable format and detectable as artificially generated or manipulated”. Article 50 has applied since 2 August 2026, the AI Act’s general date of application under Article 113. The Digital Omnibus on AI, Regulation (EU) 2026/1744, added Article 111(4), which gives the providers of systems first made available in the EU before that date until 2 December 2026 to comply with the marking duty.

Article 50(2) asks for technical solutions that are effective, interoperable, robust and reliable “as far as this is technically feasible”, taking account of the type of content, the cost and the generally acknowledged state of the art. The marking duty does not apply to systems that perform an assistive function for standard editing or that do not substantially alter the input data.

To help providers meet the duty, the Commission published a final Code of Practice on transparency of AI-generated content on 10 June 2026, and on 8 July 2026 it concluded that the code adequately covers Article 50(2), (4) and (5). Signatories to the code commit to mark the audio, images and video their systems produce, and any text they produce inside a document file, with at least two layers: digitally signed metadata and an imperceptible watermark. The commitment lasts for as long as a single technique falls short of Article 50(2)’s requirements on its own. For free-form text, such as a chatbot’s reply, which does not travel inside a file that can hold metadata, the code treats a watermark alone as enough.

The Commission’s guidelines on Article 50, whose content it approved on 20 July 2026, say that providers may rely on one marking technique or several, so long as the overall solution meets the requirements of Article 50(2). Signing the code is one way for a provider to show that its marking is adequate, and a provider that does not sign it has to show that by other means. Metadata can be stripped and watermarks weakened, so a provider that relies on one technique alone may find that showing harder to make.