DeepInversion (data-free via batch-norm statistics)

The idea

Distillation needs inputs to show the teacher. When the training images are gone, DeepInversion makes them from the teacher itself. Every batch-norm (BN) layer stores the running mean and variance of its input over the training set. Optimize random-noise images until the features they cause match those stored statistics, while the teacher predicts a chosen class, and you get images that look like training data to the network.

How it works

1. Make a batch x of random images a trainable tensor; pick target classes y.
2. Run the frozen teacher in eval mode. At each BN layer l, hook its input and take the per-channel batch mean mu_l(x) and variance var_l(x).
3. Loss: CE(teacher(x), y) + a_bn * Σ_l (||mu_l(x) - running_mean_l|| + ||var_l(x) - running_var_l||) + a_tv * TV(x) + a_l2 * ||x||². TV, total variation, sums differences between neighbouring pixels and keeps images smooth.
4. Update only x with Adam for a few thousand steps, randomly jittering and flipping it each step.
5. Repeat for many batches, then distill the student on the synthetic pool.

x = torch.randn(B, 3, H, W, requires_grad=True)
feats = []
for bn in [m for m in teacher.modules() if isinstance(m, nn.BatchNorm2d)]:
    bn.register_forward_hook(lambda m, i, o: feats.append((m, i[0])))
opt = torch.optim.Adam([x], lr=0.1)
for step in range(2000):
    feats.clear()
    loss = F.cross_entropy(teacher(jitter(x)), y)
    for bn, f in feats:
        mu, var = f.mean((0, 2, 3)), f.var((0, 2, 3), unbiased=False)
        loss = loss + a_bn * ((mu - bn.running_mean).norm() + (var - bn.running_var).norm())
    loss = loss + a_tv * total_variation(x) + a_l2 * x.pow(2).mean()
    opt.zero_grad(); loss.backward(); opt.step()

The code, line by line

invert_teacher in lab/experiments/transfer.py is the toy version without the BN term. X = rng.standard_normal(...) starts from noise, c holds random target classes and Y their one-hot rows. Adam([X], lr=lr) optimizes the inputs, not the weights. Each step, dz = (softmax(z) - Y) / n is the gradient of the mean cross-entropy with respect to the logits, t.backward(cache, dz) carries it back to the input as dX, and opt.step([dX + l2 * X / n]) adds the gradient of the L2 prior (the a_l2 term above). The toy MLP has no BN layers, so the statistics term, the part that makes DeepInversion work, is missing.

What the simulations show

Toy-MLP analogue, not YOLO: transfer_set (lab/results/transfer_set.json). A student distilled on 10,000 inverted inputs reached accuracy 0.667, against 0.6601 on Gaussian noise, 0.6662 on uniform noise and 0.7996 on the real 8k labeled set (teacher 0.9139). Without a BN prior, inversion barely beats noise here.

When it breaks

• It needs unfused BN layers. server.py fuses BN into the convs during warm-up, so start from a fresh YOLO("yolov8n.pt").
• Matching statistics does not guarantee diverse images; batches can collapse onto a few patterns per class.
• A detector has no single class output for CE. You would need a detection target, such as raising the bird score at chosen grid cells.

Going further

Missing here: a torch inversion loop with BN hooks, a detection-shaped target loss, and a detector distillation trainer (see detector_kd). The closest analogue is invert_teacher.

Source code

This method doesn't run here. Below is the closest code the lab does run, read from the repository when this page was built.

def invert_teacher(t: MLP, n: int, steps: int = 300, lr: float = 0.05, l2: float = 0.05,
                   seed: int = 0) -> np.ndarray:
    """Data-free distillation: synthesize inputs by gradient descent *on the
    input* so the teacher confidently predicts a random class, with an L2
    prior keeping them near the data scale. No real example is used."""
    rng = np.random.default_rng(seed)
    X = rng.standard_normal((n, t.sizes[0])).astype(np.float32)
    c = rng.integers(0, t.sizes[-1], n)
    Y = np.eye(t.sizes[-1], dtype=np.float32)[c]
    opt = Adam([X], lr=lr)
    for _ in range(steps):
        z, _, cache = t.forward(X, cache=True)
        dz = (softmax(z) - Y) / n
        _, dX = t.backward(cache, dz)
        opt.step([dX + l2 * X / n])
    return X

Simulation figures

Distilling without (much) real data

Which inputs carry the teacher's knowledge best: real labeled, real unlabeled, augmented, pure noise, or inputs dreamed up by inverting the teacher?

Full table: Distilling without (much) real data

Papers

• Yin et al. Dreaming to Distill (2020)