Tech Briefing. Your daily dose of tech news — signal, not noise.
Learning Center
One technology, explained in plain English
DTBRIEF

Model Distillation, Explained: How a Small AI Learns Everything a Big One Knows

A big AI model knows more than its answers show. Distillation copies that hidden knowledge into a small model — and it is now at the center of the US-China chip fight.

A large glowing teacher model pouring streams of light into a small student model, the light carrying patterns and shapes between them, on a deep navy background with cyan and warm yellow accents
Distillation pours a big model’s knowledge into a small one — not by copying its weights, but by teaching it from the teacher’s answers.
Teacher, not data
A student learns by imitating a big model’s answers, not by discovering patterns from raw data
Dark knowledge
The teacher’s probabilities for wrong answers carry the real knowledge — which answers look similar to which
146 → 74
Errors a small student made after distillation, nearly matching a teacher more than twice its size

1 What it is

Model distillation is a way to shrink a big AI model into a small one without losing most of what it knows. You train a small student model to imitate a large teacher model — not by copying its weights, but by learning from its answers.

The idea was formalized by Geoffrey Hinton, Oriol Vinyals and Jeff Dean in 2015, in a paper called “Distilling the Knowledge in a Neural Network.” Their insight: a trained model’s knowledge is not just in its weights, or even in its top answer. It is in the probabilities it assigns to every possible answer — including the wrong ones.

Think of it this way. A normal answer key says “Question 5, answer B.” A good teacher explains why B is right, why C is a tempting wrong answer, and why A is far off. Distillation gives the student the teacher’s explanations, not just the answer key.

2 How it works: the four steps

Every distillation runs the same four steps:

1. Teach — train the big teacher, or take one that already exists. 2. Ask — let the teacher answer a large set of examples, called the transfer set. 3. Soften — raise the temperature on the teacher’s answers. 4. Learn — train the small student to match those softened answers, alongside the true labels when they exist.

The magic is in step three. A confident teacher might give the right answer 99 percent probability and everything else almost nothing — the extra information is buried. Raising the temperature flattens the distribution, so the small differences between wrong answers become visible and learnable. Hinton called that hidden information dark knowledge.

The student is trained to match both the teacher’s softened answers and, when available, the true labels. On a classic test, a small student trained this way cut its errors from 146 to 74 — nearly matching a teacher more than twice its size, which made 67 errors.

3 Distillation vs training from scratch

Training from scratch means a small model learns from raw data and hard labels: one right answer per example, everything else zero. It works, but the model has to discover every pattern itself, and that takes enormous amounts of data and compute.

Distillation skips the discovery. The student starts from the teacher’s finished understanding, so it needs far less data and compute to reach the same level — and the finished student is small enough to run on cheap hardware.

Training from scratchLearns from raw data and one-hot labels. Needs huge datasets and compute to discover every pattern itself.
DistillationLearns from the teacher’s full probability distribution. Reaches near-teacher skill with far less data and a much smaller model.
A hard label says what is right. A soft label says how right everything else is — and that is the knowledge.

4 Why everyone’s distilling now

Three stories in today’s briefing are the same story. Reuters found Chinese labs using distillation at industrial scale to train domestic rivals from American models, after reviewing more than 300 documents, and Beijing is encouraging it to make AI cheaper across manufacturing. Washington calls it an intellectual-property risk and says it undercuts the chip curbs meant to slow China’s AI build-out.

A factory floor of small AI models being filled from one large teacher model above, with a map of the world behind them
Distillation is now industrial: whole countries and companies use it to build smaller models from big ones — and to get around chip controls.

And distillation is not only a geopolitical story. It is how the industry ships small, cheap models: DistilBERT kept 97 percent of BERT’s capability at 40 percent of the size, Microsoft’s Phi models are built on textbook-quality distilled data, and DeepSeek’s distilled reasoning models put frontier-style reasoning on ordinary hardware. Every lab that wants its model on a phone, a laptop, or a cheap server is distilling.

5 The honest part

First, a student cannot outgrow its teacher. Distillation transfers what the teacher knows — including its mistakes, its biases, and its blind spots. If the teacher is wrong, the student learns to be wrong in the same way, confidently.

Second, distillation is not free. The student needs the teacher to answer thousands or millions of examples, and running a giant teacher that many times is expensive — which is why the practice has become a legal fight. Model providers write their terms to restrict using their outputs to train rivals, and Washington has accused Chinese companies of doing exactly that at industrial scale.

The honest summary: distillation is the cheapest way to make a good model small, and the most argued-over way to make a small model good. It works because a model’s answers carry more knowledge than its labels — and that knowledge, once copied, is very hard to take back.

Key terms

  • Teacher The large, already-trained model whose answers the student learns from.
  • Student The smaller model being trained to imitate the teacher, cheap to run once trained.
  • Soft labels The teacher’s full probability distribution over all possible answers, not just the top one.
  • Dark knowledge The information hidden in the relative probabilities of wrong answers — which classes look similar to which.
  • Temperature A number that softens the teacher’s distribution so the dark knowledge becomes visible and learnable.
  • Transfer set The examples the teacher answers so the student has something to learn from.
From today’s briefing

This lesson comes from today’s Daily Tech Briefing (Episode 36, November 17, 2026): California sues OpenAI, the chip cloud loophole closes, and Nvidia’s big test. Read the full briefing: California Sues OpenAI — Daily Tech Briefing.

Sources

Hinton, Vinyals & Dean, “Distilling the Knowledge in a Neural Network” (arXiv:1503.02531, 2015) — soft labels, temperature, dark knowledge; Buciluă et al., “Model Compression” (KDD, 2006) — the original teacher-student idea; Sanh et al., “DistilBERT, a distilled version of BERT” (2019) — 40% smaller, 97% of the capability; Reuters, “China taps US AI models to train homegrown rivals as chip curbs bite” (November 17, 2026) — industrial-scale distillation and the US response.