Lesson 9 of 10
Low-rank factorization with the SVD
Any matrix W (m x n) can be written as U diag(S) V^T with singular values S sorted from large to small. Keeping only the first r gives the best rank-r approximation in squared error (the Eckart-Young theorem), and the error is exactly the energy you dropped:
||W - W_r||_F = sqrt(S[r]^2 + S[r+1]^2 + ...)
Store it as two thin matrices A = U_r * sqrt(S_r) (m x r) and B = sqrt(S_r) * V_r^T (r x n): m*r + r*n numbers instead of m*n. It only saves space when r < m*n / (m + n).
Implement factorize(W, r) returning (A, B) with A @ B the best rank-r approximation.
Your code
Tab indents. Press Esc, then Tab, to leave the editor. Ctrl or Cmd + Enter checks your code.
Pigeon's hint, 1 of 2
np.linalg.svd(W, full_matrices=False) returns U, S (a vector), Vt.
Pigeon's hint, 2 of 2
Splitting sqrt(S) between the two factors keeps them balanced in size.
import numpy as np
def factorize(W, r):
U, S, Vt = np.linalg.svd(W, full_matrices=False)
root = np.sqrt(S[:r])
return U[:, :r] * root, root[:, None] * Vt[:r]
Pigeon's tip
Pigeon's warning
Pigeon's note
Where this is used
- Low-rank factorization (SVD)Runs hereReplace a big matrix by the product of two thin ones, keeping its top singular directions.
- QLoRA (low-rank adapters on a quantized model)Runs hereFreeze the quantized model and train tiny low-rank corrections on top. Fine-tuning that costs a fraction of the memory.