How a machine learns from examples

Learning from examples

A trainee in a fraud team might learn to spot a forged invoice by working through a pile of past invoices that an experienced colleague has already marked as genuine or forged. The trainee guesses, checks the mark, and adjusts their sense of what a forgery looks like, until most guesses come out right. Machine learning, the name for building a computer program from examples, works in a similar way, only far faster and with far more of them, as described in What AI is, and what it is not, a Beginner module.

The examples are called the training data: inputs paired with the answers the system should give, such as emails marked spam or legitimate, or scans of handwriting paired with the words written. The program being built is called the model, and it holds a large set of internal numbers that together decide which answer it gives to each input and how strongly it leans towards that answer. Training shows the model an example and compares its answer with the right one. Training nudges those numbers a little whenever the answer is wrong, or right but with only a weak lean, so that the model would lean a little further towards the right answer if it saw the same example again. Repeated millions of times over the whole collection, the nudges add up and the model’s answers improve.

By the end of training, the numbers the model ends up with amount in effect to a pattern for sorting each new input into one of the answers the examples were marked with. In a spam filter, that pattern is whatever combination of features best separates the emails marked spam from those marked legitimate in the collection the filter was given. Developers, the people who build and train a model, do not write that pattern down, since training draws it out of the examples alone. Much of their influence over what a model learns therefore lies in the examples they choose to give it.

The test it has not seen

Good answers on the training data prove little, since a model can do well on those examples by remembering them. A student who memorised last year’s exam answers would likewise score full marks on last year’s paper and struggle on this year’s. Developers therefore set aside part of their examples, called the test set, and keep them out of training altogether. The model’s score on the test set, meaning the share of those examples it answers correctly, shows how it is likely to do on new inputs, since it did not learn from them. A lawyer shown an accuracy score for an AI system can therefore usefully ask whether the score was measured on a test set or on the training data.

Remembering the examples is the extreme form of a wider failure that a test set exposes, called overfitting, in which a model fits its particular examples so closely, quirks and all, that it does worse on new inputs. A trainee who learned that every forged invoice in the pile happened to use a blue letterhead would spot those forgeries perfectly and miss the next forger who chose green. The aim of training is the opposite, called generalisation, which means good performance on inputs the model was not shown in training.

Developers usually keep a third set as well, the validation set, which they consult while building the model to tune the training. Tuning means choosing the settings that training does not choose for itself, such as how large each nudge is and how long training runs, by trying several settings and keeping the ones that score best on the validation set. The validation set also shows overfitting early, since the model’s score on it typically starts to fall while its score on the training data is still rising. The settings were picked for doing well on the validation set, so by the end of development its score is not an independent check of the finished model. The test set is therefore kept back until the end, and its score gives a fair final measure only while its examples have stayed out of training.

Some test sets, which developers call benchmarks, are published so that models from different developers can be compared on the same test questions. Models trained on much of the public internet sometimes take in those published questions with the rest of their training data, which inflates their scores on those tests, a problem examined in Looking inside and testing, an Advanced module.

Confidently wrong

A trained model produces an answer for every input, together with a figure for how strongly it leans towards that answer, which looks like confidence and is often called a confidence score. A spam filter, for example, gives each email a figure such as 94 per cent spam, and a system that sorts photographs into cats and dogs gives each photograph a figure for each animal.

The confidence score is the model’s own figure for how strongly the input matched the patterns it learned for each answer, so 94 per cent spam means that the email looked much more like the spam examples than the legitimate ones. On inputs like its training examples, a strong match often goes with a right answer, which makes the confidence score a rough guide there. Each confidence score remains the model’s own reading of the input, however, and on inputs unlike its training examples the score can come apart from the chance of being right.

Shown a photograph of a fox, the cat and dog system will still answer cat or dog, because its training gave it those two answers to choose from and did not give it a way to say that it does not know. The system may even give its answer a high confidence score, because a fox shares many features with a dog.

Models fail most often, and most quietly, on inputs unlike their training data. A model trained on English contracts may therefore handle a contract governed by another legal system badly, and a model trained on scanned typed letters may misread handwriting, while each reports confidence scores as high as usual. The confidence score alone gives a user little warning, so a lawyer deciding whether to rely on a model should ask which tests it passed and whether the examples it was tested on resemble the material it will meet in use.

The data shapes the result

A model learns every pattern its examples contain, including patterns its developers did not intend it to learn. A collection of past decisions therefore teaches a model to reproduce those decisions, with whatever good judgement and whatever unfairness went into them. The model also learns less about any group of people that its examples rarely include, so it is likely to make more mistakes for members of that group.

A study published in 2018 under the title “Gender Shades” showed much higher error rates for some groups of people than for others in commercial systems that classified faces by gender. Testing three commercial systems on a new collection balanced by gender and skin type, the authors found error rates of up to 34.7 per cent for darker-skinned women, while the highest error rate for lighter-skinned men was 0.8 per cent. Measured across everyone in the new collection, the same systems were right between 87.9 and 93.7 per cent of the time, overall scores that, in the authors’ view, hid the differences between groups.

The companies selling the three systems did not say what training data they had used, so the study could not show directly whether a shortage of training examples caused the errors. The authors tested one other explanation, that photographs of darker-skinned people were of poorer quality, and found the same pattern of errors in a part of the collection where the photographs were of comparable quality. Photograph quality therefore did not fully account for the errors, and the authors suggested, without seeing the training data, that the errors may be linked to facial features or ways of presenting gender that were less represented in that data.

The study also found that two widely used collections of face images for testing such systems were made up overwhelmingly of lighter-skinned people, 79.6 per cent in one and 86.2 per cent in the other. A system tested on either collection was therefore tested mostly on lighter-skinned faces, and its overall score would say little about how it did for darker-skinned people. Testing a model separately on each group of people it will be used on is how gaps between groups come to light, since a system can be right for most people and wrong for many members of one group. Asking a supplier for the results of those tests is therefore one of the plainest questions a lawyer can put about an AI system.