Lesson 8 of 10
Structured pruning: removing neurons for real
Zeroing weights doesn't make a network faster on ordinary hardware; the matrix multiply is the same size. Structured pruning deletes whole neurons, so the matrices shrink.
Deleting hidden neuron j between two layers means three edits that must agree:
- remove column
jofW1(its incoming weights), - remove entry
jofb1, - remove row
jofW2(its outgoing weights).
Miss one and the shapes no longer match. In a CNN with skip connections the same idea needs a dependency group of layers, which is what torch-pruning solves for the cockpit's YOLOv8n.
Implement keep_neurons(W1, b1, W2, keep) returning the smaller (W1, b1, W2) that keeps only the hidden neurons listed in keep, and l1_scores(W1) returning one importance score per hidden neuron (the L1 norm of its incoming weights).
Your code
Tab indents. Press Esc, then Tab, to leave the editor. Ctrl or Cmd + Enter checks your code.
Pigeon's hint, 1 of 2
Column j of W1 is W1[:, j]; row j of W2 is W2[j, :].
Pigeon's hint, 2 of 2
Index with the whole keep array at once: W1[:, keep].
import numpy as np
def l1_scores(W1):
return np.abs(W1).sum(axis=0)
def keep_neurons(W1, b1, W2, keep):
keep = np.sort(np.asarray(keep))
return W1[:, keep], b1[keep], W2[keep, :]
Pigeon's tip
Pigeon's warning
Pigeon's note
Where this is used
- Structured (neuron) pruningRuns hereDelete whole neurons so the matrices actually get smaller and faster. The choice of which neurons to delete is the whole game.
- Channel pruning with dependency groupsRuns in the cockpitIn a CNN with skip connections and concatenations, a channel can't be removed from one layer alone; every layer that shares it must lose it too.