Skip to content

PigeonColumba compressaTheme

Lesson 2 of 10

A dense layer: forward and backward

A dense (fully connected) layer with a ReLU computes a = relu(h @ W + b) for a batch h of shape (N, in), weights W of shape (in, out) and bias b of shape (out,).

Backpropagation runs the chain rule backwards. Given g = dL/da (same shape as a), you need three things:

gz = g * (z > 0), where z = h @ W + b. ReLU passes gradient only where it was active.
dW = h.T @ gz and db = gz.sum(axis=0): how the loss changes with each parameter.
dh = gz @ W.T: the gradient to hand to the layer below.

Every network in lab/nn.py is a stack of these. MLP.backward is this function in a loop.

Your code

import numpy as np


def dense_forward(h, W, b):
    """Return (a, cache): a = relu(h @ W + b); cache = whatever backward needs."""
    raise NotImplementedError


def dense_backward(g, cache):
    """Given g = dL/da, return (dh, dW, db)."""
    raise NotImplementedError

Checking your code and saving a draft need JavaScript, which is turned off in this browser. Turn it on and reload the page to check your code. You can still edit it here, and the hints and the reference solution are below.

Pigeon's hints (2)

Hint 1 of 2

Keep z (before the ReLU) in the cache; the ReLU mask is z > 0.

Hint 2 of 2

Check shapes: dW must be (in, out) like W, so it's h.T @ gz.

The reference solution

import numpy as np


def dense_forward(h, W, b):
    z = h @ W + b
    return np.maximum(z, 0), (h, W, z)


def dense_backward(g, cache):
    h, W, z = cache
    gz = g * (z > 0)
    return gz @ W.T, h.T @ gz, gz.sum(axis=0)

Where this is used