How a smaller AI learns from a larger one

A smaller AI can learn from a larger one’s answers, then work independently. Knowledge distillation moves expensive teaching into training, while the resulting student handles subsequent requests. What transfers depends on the lessons it receives.[1]

An answer can contain more than one lesson

Consider an invented email classifier choosing bill, receipt or payment reminder. Its teacher assigns 70%, 25% and 5%. Keeping only bill discards the difference between a plausible alternative and a distant one. The student can learn those distinctions. These numbers are illustrative.[1]

Hinton, Vinyals and Dean explained this advantage in their 2015 distillation paper. Training can use the teacher’s probability distribution alongside correct labels. A setting called temperature can soften that distribution so smaller probabilities carry more influence. The student learns from the pattern of alternatives as well as the winning answer.[1]

Smaller needs a specific comparison

DistilBERT gives a concrete historical example. Introduced in 2019, it reduced BERT-base’s twelve transformer layers to six and combined several training objectives, including matching teacher predictions and internal representations. Its starting parameters came from alternating teacher layers. That is more specific than copying a collection of finished answers.[2]

The smaller model had 40% fewer parameters. Its aggregate was approximately 97% of the teacher’s, which explains a widely repeated retention claim. That denominator matters: a collection of language-understanding tests supports a statement about those tests. It cannot establish that every skill, rare case or kind of knowledge survived the reduction.[2]

Parameter count describes the trained model; GLUE measures selected performance. Application speed needs separate measurement on the intended hardware, workload and software.[2]

A specialist can learn more from an explanation

The 2023 Distilling Step-by-Step study taught smaller models two related tasks: produce the answer and produce an explanation generated by a large teacher. Separate task prefixes distinguished them. Once trained, the student could be asked for the answer alone. The explanation was useful teaching material without becoming a compulsory extra output on every request. That arrangement supplies two forms of supervision for each example, rather than treating the explanation as decoration attached to an answer.[3]

On round 1 of the ANLI language-inference dataset, a 770-million-parameter T5 model trained with human labels and PaLM-generated explanations, using 80% of the training examples, outperformed the study’s few-shot chain-of-thought PaLM 540B comparison. That is a striking specialist result. It does not mean the student acquired everything the much larger teacher could do.[3]

The best solver may be a poor match for its student

Choosing a teacher introduces another question: how learnable are its answers? The 2026 GRACE paper tested fifteen teacher configurations in mathematics distillation. The teacher with the strongest own test performance did not necessarily produce the strongest student. A leaderboard ranking and a teaching ranking were measuring different relationships.[4]

GRACE examined the changes that teacher answers would induce in the student’s parameters. It assessed whether those training signals offered useful diversity and alignment, then used that information to select a teacher. The evidence comes from particular model families and mathematics tasks. Its broader implication is practical: when building a smaller specialist, testing the student after teaching tells you something that testing the teacher alone cannot. The score can be estimated from sample prompts without access to the teacher’s internals or a separate verifier.[4]

Copying an answer and learning to continue are different tasks

A 2026 ICML paper on mixing hard and soft labels investigated a subtle mismatch. During training, a student can be shown the teacher’s preceding words and learn what should follow. When working independently, it must continue after its own words. One deviation can place it in a situation absent from those lessons.[5]

The researchers found benefits from mixing completed teacher responses with probability-based supervision. In their experiments, improved task performance could accompany a worse fit to the teacher’s training distribution. The useful objective was a student that continued successfully, not simply one that reproduced that distribution more closely. Some positions demanded a precise choice; others allowed several good continuations.[5]

Another 2026 approach, SCoRe, made the student’s own attempts part of the teaching process. A teacher examined an attempted solution and corrected its earliest error. The student then continued, potentially requiring another intervention. That produced examples addressing situations the student actually encountered. The method subsequently used supervised training and reinforcement learning, including checks on final task success.[6]

This distinction changes how a result should be read. A student trained through corrections, practice and outcome-based rewards has received more than a transcript to imitate. SCoRe’s results therefore do not establish that copying alone is sufficient. Its limitations also acknowledge continuing dependence on teacher quality and the possibility of accumulating errors.[6]

For someone choosing a compact model, useful questions follow: what was it taught, which tasks were measured, and what other training followed? Distillation is most informative when the intended job, the teaching material and the student’s demonstrated performance are all visible.[6]

Report an error or suggest a correction