Field Guide
43 compression methods. 31 run in your browser and 3 in the YOLOv8n cockpit on your own machine, each shown with the source code that runs it. The other 9 are reference write-ups for kinds of models the lab doesn't have: 7 show the closest code that does run, and 2 have no code at all.
Small networks
Every method runs here on a toy 20-class problem with a known best-possible answer. Start here: the ideas are the same at every scale, and most runs finish in under a minute.
Distillation
- Knowledge distillation (Hinton)Runs hereTrain a small model to match a big model's softened output probabilities instead of (or as well as) the one-hot labels.
- Distillation loss variantsRuns hereForward KL, reverse KL, logit MSE, hard pseudo-labels, and teachers that reveal only their top-k probabilities.
- Feature & relational distillationRuns hereMatch the teacher's hidden activations (FitNets) or the similarity structure between examples (similarity-preserving KD), not just its outputs.
- Distilling without real dataRuns hereFeed the teacher noise, or inputs synthesized by gradient descent on the teacher itself, and distill on those.
- Function matching (patient, consistent teaching)Runs hereQuery the teacher on freshly augmented inputs every step and train for longer: distillation as matching a function, not a dataset.
- Teacher assistantsRuns hereWhen the student is far smaller than the teacher, distill through a mid-sized model in between.
- Ensemble teachersRuns hereAverage several teachers' predictions and distill the average into one small model.
- Self-distillation (born-again networks)Runs hereA model teaches an identical copy of itself, sometimes for several generations.
- The perfect teacherRuns hereDistill from the exact Bayes posterior. An upper bound on what any teacher's soft labels can give a student of a given size.
Pruning
- Magnitude pruningRuns hereSet the smallest weights to zero. Cheap and surprisingly effective, as long as you let the network recover afterwards.
- Structured (neuron) pruningRuns hereDelete whole neurons so the matrices actually get smaller and faster. The choice of which neurons to delete is the whole game.
- Lottery ticketsRuns hereTrain, prune, rewind the surviving weights to their early values, retrain. Tests whether the sparse network was trainable from the start.
Quantization
- Post-training quantizationRuns hereRound every weight to one of 2^b − 1 evenly spaced levels, symmetric around zero (two levels at 1 bit), after training. No data, no gradients; how fine the scale groups are decides how low b can go.
- Quantization-aware trainingRuns hereKeep training with quantized weights in the forward pass and full-precision weights in the backward pass (the straight-through estimator).
- Weight sharing (k-means codebooks)Runs hereEach layer keeps only k distinct values, chosen by k-means. Store a log2(k)-bit index per weight plus a tiny codebook.
Training small models
- Over-parameterize, then collapseRuns hereTrain each weight matrix as a product of two wider ones, then multiply them back together. Same network at inference, different training path.
- Regularizers & training tricksRuns hereLabel smoothing, mixup, weight averaging (EMA), longer training: what each does for a capacity-starved model.
- Train big, then prune to sizeRuns hereTo get a small network, train a wide one and cut it down instead of training the small one directly.
Specialization
- Specialist headsRuns hereKeep only the outputs the task needs (bird vs. rest), and decide how much of the teacher's 'what else could it be' to keep.
- Task-aware pruningRuns hereScore neurons by how much they matter for the one task you keep, prune the rest, collapse the head, recover by distilling only that task.
- Recovery data for a specialistRuns hereWhat the recovery set must contain, and whether images of the target alone are enough.
Language-model methods
Algorithms invented for large language models. The layer-by-layer ones (GPTQ, AWQ, SmoothQuant, Wanda, SparseGPT, QLoRA, ShortGPT) are pure linear algebra on one weight matrix at a time, so they run here on the same toy network. Methods that need text, tokens or a KV cache are explained but marked as reference.
Pruning
- WandaRuns herePrune by |weight| times the norm of its input activation, per output row. No retraining, no weight updates.
- SparseGPTRuns herePrune with GPTQ's error-compensation trick: every removed weight's error is absorbed by the weights that remain.
- Expert pruning in mixture-of-expertsReferenceDrop or merge the experts a router rarely picks for your task: structured pruning where the 'neurons' are whole sub-networks.
Quantization
- Round-to-nearest (RTN)Runs hereThe baseline every LLM quantization paper beats: round each weight to the nearest level, one scale per output row.
- GPTQRuns hereQuantize one column at a time and push each rounding error onto the columns not yet quantized, using the inverse Hessian from calibration data.
- AWQ (activation-aware weight quantization)Runs hereScale up the weight columns that see large activations before rounding, so the weights that matter most get the finest resolution.
- SmoothQuant (weights + activations)Runs hereActivations with a few huge channels can't be quantized to 8 bits; move that difficulty into the weights with an exact per-channel rescaling.
- QLoRA (low-rank adapters on a quantized model)Runs hereFreeze the quantized model and train tiny low-rank corrections on top. Fine-tuning that costs a fraction of the memory.
- KV-cache quantizationReferenceQuantize the attention keys and values stored during generation, which dominate memory at long context.
Training small models
Vision / detector (YOLOv8n cockpit)
What changes when the model is a convolutional detector like YOLOv8: channels instead of neurons, dependency groups, batch-norm. The cockpit methods run in the YOLOv8n cockpit's web UI (cockpit/server.py) on a real model; the rest are explained for reference.
Distillation
- DeepInversion (data-free via batch-norm statistics)ReferenceSynthesize images whose feature statistics match the teacher's stored batch-norm means and variances, then distill on them.
- Attention transferReferenceMatch where the teacher's feature maps are active (spatial attention maps), a feature-distillation loss that needs spatial structure.
- Distilling a detectorReferenceDetection outputs are boxes, not one distribution: distill the per-anchor class scores, the box regressions and the neck features.
Pruning
- Channel pruning with dependency groupsRuns in the cockpitIn a CNN with skip connections and concatenations, a channel can't be removed from one layer alone; every layer that shares it must lose it too.
- Prune, then fine-tune (cut and regrow)Runs in the cockpitThe cockpit's core loop: remove channels, then fine-tune the survivor on bird images until mAP climbs back.
Techniques not covered yet
Techniques this guide covers only in part, or not at all. Where the Coop applies the same number format to Pigeon's own weights, the line says how.
GGUF k-quants and the importance matrix
Not covered
No page covers k-quants, their super-blocks or the importance matrix. Their starting point, round-to-nearest with one scale per group of weights, is on Round-to-nearest (RTN) and Post-training quantization.
FP8 (E4M3 and E5M2)
Not covered
No page covers FP8 yet. In the Coop, a Change number format step converts Pigeon's weights (not its activations) to FP8 E4M3 or E5M2: open the Coop and add that step.
FP4 microscaling (MXFP4 and NVFP4)
Partly covered
Quantizing a detector uses a 4-bit float grid (E2M1) with a scale per group of 32 or 16 weights, but not the MX or NVFP4 scale formats. The Coop has both for Pigeon's weights: the MXFP4 preset, and NVFP4 as a scaling option in its Change number format step.
bitsandbytes NF4 and QLoRA
Partly covered
QLoRA (low-rank adapters on a quantized model) runs the core idea, low-rank adapters trained on a frozen quantized model, but its base is plain integers with a scale per 16 weights: NF4, double quantization and paged optimizers are explained but not implemented. Weight sharing (k-means codebooks) mentions NF4's fixed codebook next to its own learned ones.
Rotations (QuaRot, SpinQuant, ParoQuant)
Not covered
Rotation methods get one sentence each, in the further-reading notes of SmoothQuant (weights + activations) and GPTQ. Hadamard transforms and ParoQuant are not explained. SmoothQuant tackles the same outlier problem with per-channel scaling instead of a rotation.
KV-cache quantization
Partly covered
KV-cache quantization explains KIVI and KVQuant but runs nothing: the lab's toy model has no attention and no cache. SmoothQuant (weights + activations) does run the per-token activation quantizer it points to.
MLX
Not covered
MLX isn't mentioned anywhere in the lab. Its default 4-bit scheme (a scale and a zero point per 64 weights) can be rebuilt in the Coop with a Quantize to integers step at 4 bits, per group of 64, asymmetric, and its mxfp4 mode matches the Coop's MXFP4 preset. Both reproduce the number formats, not MLX itself.
2:4 sparsity
Partly covered
SparseGPT, Wanda and Magnitude pruning remove single weights with no pattern; each mentions 2:4, but none simulates it. The Coop's 2:4 sparsity preset keeps the 2 largest of every 4 weights in Pigeon, by magnitude, with no calibration and no retraining.
TorchAO, LiteRT and Core ML
Not covered
None of these toolchains is covered. Some of the schemes they apply are: int8 and int4 weights on Post-training quantization and Quantization-aware training, k-means palettes on Weight sharing (k-means codebooks), and Batch-norm folding. The Coop applies two of them to Pigeon: its INT8 per-channel preset and its 16-colour palette preset.
Covered with pages that run here: AWQ (activation-aware weight quantization), GPTQ, SmoothQuant (weights + activations); pruning followed by distillation (Magnitude pruning, Structured (neuron) pruning, Layer dropping and others in the families above); and quantization-aware distillation, the distillation option of Quantization-aware training.