How AI chooses which examples a person should teach it next

Imagine a small ceramics maker with thousands of photographs and a specialist who can identify subtle defects. The scarce resource is the specialist’s time. Asking them to label every picture might spend hours on obvious examples. An active-learning system instead chooses which pictures it should learn from next. This is a constructed business example, not a reported deployment.[1]

The cycle is straightforward: train a model on an initial labelled set, examine an unlabelled pool, request human labels for selected examples, then train again. Lewis and Gale described an influential uncertainty-sampling approach in 1994. The system seeks examples useful for improving the classifier, rather than simply collecting the records most likely to match one category.[1]

An uncertain answer can mean different things

Choosing close calls sounds sensible. But uncertainty may arise because the model lacks relevant knowledge or because the evidence itself is ambiguous. The 2017 Deep Bayesian Active Learning paper used repeated predictions with different dropout masks to approximate variation among plausible models. Its BALD selection method looked for examples whose labels could reduce uncertainty about the model.[2]

Both rows average to a coin flip, but they offer different learning opportunities. In the first, a label can help resolve competing confident judgements. In the second, even individual predictors find the example ambiguous. Bayesian active learning formalises a related distinction; ordinary statements from a chatbot about feeling unsure do not supply this measurement.[2]

Ten similar questions can teach less than ten different ones

The ceramics specialist might receive a batch of ten nearly identical pictures of the same kind of hairline crack. Each could sit close to the classifier’s decision boundary, yet answering all ten might add little beyond answering a few. This is an original illustration of batch redundancy: informative examples individually need not make the most informative collection.[3]

The 2018 core-set approach addressed batch selection by representing examples in a learned feature space and seeking a set that covered the wider pool. In that representation, similar images lie near one another. Selecting a spread of examples can expose the learner to different regions of the task. The geometry depends on what the model has learned to represent; visual diversity alone is not a complete measure of usefulness.[3]

BADGE, published at ICLR 2020, combines uncertainty and diversity through a different signal: how an example could change the model during training. Because the true label is still unknown, it temporarily uses the model’s most likely label to construct a gradient representation. That is a selection device. The person’s subsequent annotation supplies the actual training label.[4]

The method chooses a batch spread across those representations. Broadly, magnitude captures how much learning an example might prompt, while direction distinguishes different changes. This connects two sensible requirements: ask questions the learner has not settled, and avoid spending the entire batch on essentially the same lesson. It does not assume every difficult-looking example is worth the same amount of human attention.[4]

The right question also depends on who must answer it

A 2009 study of Uspanteko language annotation measured something easily lost when researchers count labelled examples: time. Its two participants were both documentary linguists, but one already knew the language and the other did not. They worked under different selection and suggestion conditions. The study compared progress by elapsed annotation time as well as by the amount of text labelled.[5]

Uncertainty sampling worked best for the participant with Uspanteko experience. For the other participant, random selection with suggested labels worked best. Two annotators cannot establish a universal rule about human expertise. They do show why a difficult example’s learning value and its human cost need to be measured together. Saving examples does not automatically save minutes.[5]

A 2026 EACL paper brought unequal costs directly into the selection rule. Its Knapsack-BALD method first identified informative candidates, then chose among them subject to a total budget. The evaluation covered eleven African languages under simulated cost schemes, including difficulty-based proxies. These were computational experiments, not measurements of a paid annotation team’s productivity.[6]

An invented budget makes the distinction concrete. With twelve minutes available, six two-minute items fit. An eight-minute item leaves room for only two of those shorter items. The expensive example might still be the better purchase if it teaches a valuable distinction. A system needs some estimate of that benefit and cost; merely requesting the hardest available cases does not solve the allocation problem.[6]

Measure what the expert’s time actually bought

Evidence for sophisticated selection must also survive a good simple comparison. A CVPR 2022 study revisited neural active-learning experiments and found that gains over random sampling depended strongly on training choices, regularisation and experimental controls. Strengthening the learning pipeline could narrow the apparent advantage. Its results caution against treating a selection algorithm’s headline saving as a property that transfers unchanged to every dataset.[7]

For the ceramics maker, a meaningful comparison would give active selection and random selection the same annotation-time budget, train comparable models and judge both on a separate, representative set of products. Several runs would help reveal how much depends on the starting examples. Those are proposed evaluation choices, not a test performed here. The desired result is better decisions for the human effort spent—and evidence showing where the gain came from.[7]

Report an error or suggest a correction