Post-training quantization

The idea

A trained network stores 32-bit floats but rarely needs that much precision. Post-training quantization (PTQ) rounds each weight to one of a small set of evenly spaced levels, so it can be stored as a b-bit integer plus a shared scale (the float that turns integers back into weights). It needs no data and no training. How low b can go depends on how many weights share a scale. One scale per matrix is set by its largest weight and leaves few levels for typical ones. A scale per group of 16 weights tracks local size much more closely.

How it works

1. Flatten the weight matrix and cut it into groups of g consecutive values (or keep one group for the whole tensor).
2. For b >= 2 bits, symmetric quantization uses the integers -qmax..qmax with qmax = 2^(b-1) - 1. For 4 bits that is -7..7: 15 levels.
3. Per group: scale = max|w| / qmax, then q = clip(round(w / scale), -qmax, qmax).
4. The stored model is q (b bits each) plus one scale per group. The weight used at inference is q * scale.
5. For b = 1 there are just two levels: w_hat = sign(w) * mean|w| per group.

qmax = 2 ** (bits - 1) - 1
scale = abs(group).max() / qmax
w_hat = clip(round(group / scale), -qmax, qmax) * scale

The code, line by line

fake_quantize in lab/nn.py is a line-for-line numpy twin of harness/quant.py:fake_quantize, the function the YOLOv8n cockpit runs on its detector. Both return floats ("fake" quantization) holding exactly the values a b-bit format could store, so measured accuracy matches a real export.

if bits >= 16: return w is the native setting. flat = w.reshape(-1) flattens row by row. lab/nn.py stores W as (inputs, outputs), so a group of 16 is 16 weights leaving one input towards 16 consecutive neurons. With no group_size, g = flat[None, :] is one group. Otherwise pad = (-n) % group_size zeros are appended and reshape(-1, group_size) makes one row per group.

For 1 bit, scale = np.maximum(np.abs(g).mean(1, keepdims=True), 1e-12) and q = np.where(g >= 0, 1.0, -1.0), so even an exact zero becomes +scale. The torch version does the same. For more bits, qmax = (1 << (bits - 1)) - 1, scale = max|g| / qmax, and q = np.clip(np.round(g / scale), -qmax, qmax). The 1e-12 floor guards all-zero groups. out = (q * scale) is reshaped, unpadded and cast back to float32.

quantize in lab/compress.py copies the net, sets net.qbits, net.qgroup, and bakes the result in with net.W = [net.eff(i) for i in range(net.n_layers)], since MLP.eff applies fake_quantize when qbits is set. Clearing qbits leaves an ordinary float MLP whose weights lie on the grid.

One consequence is easy to miss. At 2 bits, qmax = 1, so the levels are only {-scale, 0, +scale}, and every weight smaller than half the group maximum becomes zero. The params column counts nonzero weights and shows this directly.

What the simulations show

From quantization, accuracy on 20 classes. The "teacher" is the 3x256 net (140,052 params). The "student" is a 2x32 net (2,132 params) distilled from it.

8 and 6 bits are free for both models. Below that, group scales are worth a lot (teacher at 3 bits: 0.7766 → 0.8970), and the small student degrades much earlier than the big teacher. Per-tensor 2-bit is worse than 1-bit. At 2 bits only 3,616 of the teacher's 140,052 weights stay nonzero, while 1-bit keeps every weight as ±mean.

Scales cost storage. Counting an fp16 scale per group, the teacher at 4 bits takes 582.3 kbit per-tensor and 721.5 kbit with group-16, against 4,481.7 kbit in fp32. A group of 16 adds one bit per weight, and a group of 4 adds four.

Page runs: the default (teacher, 4-bit, group 16) gives 0.9135. Group 4 at 2 bits gives 0.8278, but with four extra bits of scale per weight.

The cockpit's carve.py on YOLOv8n found the same shape with a lower floor: 8-bit about free, 6-bit alive only with group scales (group size 128), 4-bit collapsed. The small student, with little spare capacity, is the closer analogue.

When it breaks

• Outliers. One large weight sets scale for its whole group and pushes everything else towards zero. This is why per-tensor scales fail first.
• 2-bit symmetric is ternary. Most weights become zero, which is effectively heavy pruning.
• Group overhead. Very small groups cost more in scales than they save in bits.
• Activations are not quantized here. Real integer inference also rounds activations, which has its own outlier problem (see SmoothQuant).

Try this

• Set scale shared by to tensor and read the bits chart. Then switch to 16. Find the lowest bits that stays within 0.01 of fp32 in each case.
• Set bits to 2 and compare the params column for tensor and 4. The nonzero count shows how many weights rounded to zero.
• Switch model to student and repeat the first experiment. With group 16 the lowest safe width moves up two bits, from 4 to 6; per tensor it is 6 for both models.

Going further

Real deployments store integers (int8, int4) with per-channel or per-group fp16 scales, and quantize activations too, using a calibration set to pick their ranges. At 4 bits and below, calibration-based methods (GPTQ, AWQ) beat round-to-nearest, and QAT beats both when you can afford training. In the YOLOv8n cockpit, the Precision module applies harness/quant.py:fake_quantize to every Conv2d (via harness/model.py), so accuracy measured in the cockpit is what an export with the same scheme would give.

Source code

The exact code that runs, read from the repository when this page was built.

def fake_quantize(w: np.ndarray, bits: int, group_size: int | None = None) -> np.ndarray:
    """NumPy twin of ``harness.quant.fake_quantize`` (same semantics)."""
    if bits >= 16:
        return w
    flat = w.reshape(-1)
    n = flat.size
    if group_size is None or group_size <= 0 or group_size >= n:
        g, pad = flat[None, :], 0
    else:
        pad = (-n) % group_size
        g = np.concatenate([flat, np.zeros(pad, w.dtype)]).reshape(-1, group_size)
    if bits == 1:
        scale = np.maximum(np.abs(g).mean(1, keepdims=True), 1e-12)
        q = np.where(g >= 0, 1.0, -1.0)
    else:
        qmax = (1 << (bits - 1)) - 1
        scale = np.maximum(np.abs(g).max(1, keepdims=True), 1e-12) / qmax
        q = np.clip(np.round(g / scale), -qmax, qmax)
    out = (q * scale).reshape(-1)
    if pad:
        out = out[:-pad]
    return out.reshape(w.shape).astype(w.dtype)

def quantize(net: MLP, bits: int, group: int | None = None) -> MLP:
    """Post-training quantization: bake fake-quantized weights in."""
    net = net.copy()
    net.qbits, net.qgroup = bits, group
    net.W = [net.eff(i) for i in range(net.n_layers)]
    net.qbits = None
    return net

def fake_quantize(
    w: torch.Tensor,
    bits: int,
    group_size: int | None = None,
) -> torch.Tensor:
    """Return a fake-quantized copy of ``w`` at ``bits`` bits.

    - ``bits >= 16``: returned unchanged (top of the dial / native).
    - ``bits == 1``: binary sign scheme, ``w_hat = sign(w) * mean(|w|)`` per group.
    - ``2..15``: symmetric linear quant with a per-group scale.

    ``group_size`` is the number of consecutive elements (along the flattened
    tensor) that share one scale. ``None`` -> the whole tensor shares one scale.
    Smaller groups = finer scales = less error (but more scale metadata in a
    real format). ``group_size == 1`` is a deliberate no-op.
    """
    if bits < 1:
        raise ValueError("bits must be >= 1")
    if bits >= 16:
        return w.clone()

    orig_shape = w.shape
    orig_dtype = w.dtype
    flat = w.reshape(-1).float()
    n = flat.numel()

    if group_size is None or group_size <= 0 or group_size >= n:
        groups = flat.unsqueeze(0)  # 1 x N: one shared scale
        pad = 0
    else:
        pad = (-n) % group_size
        if pad:
            flat = torch.cat([flat, flat.new_zeros(pad)])
        groups = flat.reshape(-1, group_size)  # G x group_size

    if bits == 1:
        scale = groups.abs().mean(dim=1, keepdim=True).clamp_min(1e-12)
        q = torch.sign(groups)
        q = torch.where(q == 0, torch.ones_like(q), q)  # map exact zeros to +1
        deq = q * scale
    else:
        qmax = _pos_levels(bits)
        scale = groups.abs().amax(dim=1, keepdim=True).clamp_min(1e-12) / qmax
        q = torch.clamp(torch.round(groups / scale), -qmax, qmax)
        deq = q * scale

    out = deq.reshape(-1)
    if pad:
        out = out[:-pad]
    return out.reshape(orig_shape).to(orig_dtype)

Simulation figures

Quantization: PTQ vs. QAT, labels vs. KD, big vs. small

How few bits per weight survive? Does a per-group scale rescue low bits, does quantization-aware training rescue more, and do small models tolerate it worse?

Full table: Quantization: PTQ vs. QAT, labels vs. KD, big vs. small

Papers

• Jacob et al. Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference (2018)