Lesson 7 of 10

Fake quantization with group scales

To store a weight in b bits with a symmetric scheme, pick a scale s and keep the integer q = round(w / s), clipped to [-qmax, qmax] where qmax = 2^(b-1) - 1. The stored model holds q and s; using it means computing q * s.

Fake quantization returns q * s in float directly, so you can measure the accuracy of the quantized model without any integer kernels. That's what harness/quant.py does to YOLO and lab/nn.py:fake_quantize does here.

The scale is shared by a group of consecutive weights: s = max|w in group| / qmax. Smaller groups, more scales, less error.

Implement fake_quantize(w, bits, group) for bits in 2..8, where group=None means one scale for the whole tensor. Group along the flattened tensor; the last group may be shorter.

Your code

import numpy as np


def fake_quantize(w, bits, group=None):
    """Return an array like w, rounded to the levels a `bits`-bit symmetric code can store."""
    raise NotImplementedError

Checking your code and saving a draft need JavaScript, which is turned off in this browser. Turn it on and reload the page to check your code. You can still edit it here, and the hints and the reference solution are below.

Pigeon's hints (2)

Hint 1 of 2

Flatten, loop over chunks of group, quantize each with its own scale, reshape back.

Hint 2 of 2

group=None is the same as one group the size of the whole tensor.

The reference solution

import numpy as np


def fake_quantize(w, bits, group=None):
    qmax = 2 ** (bits - 1) - 1
    flat = w.ravel().astype(float)
    g = group or flat.size
    out = np.empty_like(flat)
    for s in range(0, flat.size, g):
        blk = flat[s:s + g]
        scale = max(np.abs(blk).max(), 1e-12) / qmax
        out[s:s + g] = np.clip(np.round(blk / scale), -qmax, qmax) * scale
    return out.reshape(w.shape)

Your code

Where this is used