Words as numbers

Cutting text into tokens

Before a language model can work on a piece of text, software called a tokeniser cuts the text into tokens, since the model can process only numbers and each token has to be turned into one. Tokens are common whole words, pieces of rarer words, punctuation marks or single bytes, the small units in which a computer stores characters. The tokeniser replaces each token with its number in the vocabulary, a fixed list of every token the model knows, so the model receives a sentence as a short row of whole numbers.

AI services are priced and limited in tokens, so a lawyer reading their terms needs a rough way of turning tokens into words. OpenAI’s documentation gives a rough rule of thumb for English text: one token is about four characters, or three quarters of a word, so 100 tokens come to roughly 75 words.

The dominant method of building a vocabulary is byte-pair encoding, which began as a way of compressing data before researchers adapted it to splitting words for machine translation. Byte-pair encoding starts with every word in a large body of text spelled out letter by letter, finds the pair of neighbouring pieces that occurs most often and merges that pair into one longer piece, which joins the vocabulary as a new token. The count and the merge are repeated thousands of times over, until the vocabulary holds the number of tokens its designers chose. Frequent words therefore end up as single tokens and rare words are spelled out in pieces, so a common word such as “the” costs one token where a rare surname or technical term may cost several.

Vocabularies have grown with each generation of models, from about 50,000 tokens in GPT-2 to about 200,000 in the tokeniser behind GPT-4o and the OpenAI models that followed it, up to and including GPT-5. Larger vocabularies let more words count as single tokens, including words of languages other than English: Meta’s Llama 3, for instance, took 100,000 tokens from an existing tokeniser and added 28,000 to support those languages, for 128,000 in all. Tokenisers differ between model families, so the same document can come to a different number of tokens on different models, and a price per million tokens quoted by one provider may not compare directly with another’s.

Where tokens trip models up

A model receives each token only as its number in the vocabulary, which it then swaps for a list of numbers, so the letters or digits inside the token are not shown to it. Tasks that depend on those letters or digits can therefore go wrong in ways that surprise a person.

Arithmetic shows the effect most plainly, since a tokeniser may split a number into chunks that cut across place value, the units, tens and hundreds a person lines up to do a sum. The tokeniser behind GPT-4, for example, groups digits into chunks of up to three read from the left, so 12345 becomes “123” followed by “45”. Researchers found that inserting commas into numbers, which forces the digits to be grouped from the right, improved the arithmetic of GPT-3.5 and GPT-4. With a comma, the tokeniser reads 12,345 as “12” and “345”, chunks that line up with place value. A total that a model works out, such as the sum of a schedule of losses, therefore deserves a check by other means.

Counting the letters in a word is a familiar weakness of language models, usually explained by the model receiving a token number without seeing the letters inside it. Models can still learn which letters make up each token from the text they are trained on, much as a person learns a spelling by heart. A preprint, a paper posted before peer review, found that models could pick out the letters in a word and still miscount them, and that their mistakes were more frequent in words where a letter appears more than once. The pattern suggests that difficulty with counting repeated items plays a part as well, alongside the way text is cut into tokens.

A vocabulary built from a body of text that is mostly English merges the pairs of letters common in English, so common English words end up as whole tokens. Other languages are left in more and smaller pieces, and the same meaning therefore needs more tokens. One study found that the same text, translated, can take up to 15 times as many tokens in one language as in another. Another study tested OpenAI’s programming interface, the paid service through which businesses send text to its models, in 22 languages and found that speakers of many of them pay more under per-token pricing while getting poorer results. More tokens per sentence also means that less of their text fits into the context window, the amount of text a model can take in at once, so a long document in such a language may need cutting up where its English translation would fit whole.

Embeddings: meaning as position

After the cutting, each token number is swapped for a list of numbers, called a vector or an embedding, by looking up its row in a table filled in during training, the long process in which a model learns by adjusting its numbers over large amounts of text. Each list can be read as the coordinates of a point, like a map reference, on a map with thousands of dimensions where a paper map has two, east to west and north to south. In GPT-3, for example, each token becomes a list of 12,288 numbers, one coordinate for each of 12,288 dimensions. Training places tokens that appear in similar contexts near one another on that map, so closeness on the map comes to stand for closeness in meaning.

The idea behind the map goes back to the 1950s, when linguists observed that words used in similar contexts tend to have similar meanings, an observation now called the distributional hypothesis. “Court” and “tribunal”, for instance, turn up beside many of the same words, such as “judge”, “hearing” and “appeal”, so training places them close together.

Some steps across the map also come to stand for consistent relationships: the step that leads from the point for “man” to the point for “woman” is roughly the same as the step from “king” to “queen”. The step from “man” to “woman” is found by subtracting the coordinates of “man” from those of “woman”, so adding that step to “king” is written as king minus man plus woman. The arithmetic yields a point on the map, and a search then picks the word whose point lies nearest, which in this case is “queen”. The standard search leaves out king, man and woman, the three words the puzzle started from. Researchers reported this pattern in 2013, in two papers from overlapping teams, one of which introduced word2vec, a system built at Google that learns such a map quickly from large amounts of text.

The king and queen example needs careful reading, since it holds only because the search leaves out king, man and woman themselves. A 2020 study tested what happens when those three words are allowed as answers, using the standard analogy test, a set of puzzles built like the king and queen one. The share of correct answers fell from 0.71 to 0.21, or from about seven in ten to about two in ten, with the king minus man plus woman arithmetic. In most of the failures the nearest point was the starting word, “king” in the famous example. Even with the three words left out, the best model in the 2013 word2vec paper answered only about two thirds of its analogy questions correctly. A striking demonstration can therefore overstate how well a technique works, which is a reason to ask what a vendor’s or an expert’s headline example leaves out.

Embeddings learned from human text also absorb its biases: one study found that word embeddings hold the associations about race and gender, among others, that psychological tests find in people. Researchers have also used analogy puzzles to look for bias, by adding the step from “man” to “woman” to a word such as “doctor” and reading a stereotyped answer as a sign of bias in the map. The 2020 study argued that analogy tests are an inaccurate way of measuring those biases. The standard search will not give back “doctor” itself, even where “doctor” is the nearest word, so it returns the next nearest word, which may be a stereotyped one such as “nurse”. A bias audit that relies on analogy tests may therefore give a distorted picture.

The maps in those studies came from older systems such as word2vec, which gave each word one fixed vector, so “bank” had the same vector beside a river and beside a loan. A transformer, the design behind modern language models described in The transformer, an Intermediate module, starts from a fixed vector for each token and revises it at each of a series of processing stages, called layers, using the words that come before it. The revised vector lets a modern model read “consideration” in a contract differently from “consideration” in a letter of thanks.