Backpropagation

Recap

In Chapter 2, we built a computation graph that records operations and stores forward-computed values.

The Problem

We have a graph where the output e depends on inputs a, b, c. We want to know: how much does e change if we nudge a slightly? That's $\frac{\partial e}{\partial a}$ — the gradient of e with respect to a.

We could compute this numerically (perturb a by a tiny $\epsilon$ , recompute, divide), but that requires one forward pass per parameter. With millions of parameters, that's absurdly slow.

Backpropagation computes all gradients in a single backward pass.

Topological Ordering

To backpropagate, we need to process nodes in the right order. If c = a * b, we must know c's gradient before we can compute a's and b's gradients.

This means we traverse the graph in reverse topological order: from output to inputs. A depth-first search visits children first, then appends the current node. When we reverse this list, every node appears after all nodes that depend on it.

Gradient Rules

For each operation, we derive the local gradient using calculus, then multiply by the upstream gradient (chain rule):

Operation	If $v = \text{op}(...)$	Gradient rule
Add $v = a + b$	$\frac{\partial v}{\partial a} = 1$	`grad[a] += grad[v]`
Mul $v = a \times b$	$\frac{\partial v}{\partial a} = b$	`grad[a] += grad[v] * data[b]`
Tanh $v = \tanh(a)$	$\frac{\partial v}{\partial a} = 1 - v^2$	`grad[a] += (1 - v²) * grad[v]`

The += is critical — it accumulates gradients. A node might be used in multiple places, and each usage contributes to the total gradient.

The Backward Pass

We start by setting the root's gradient to 1.0 (the derivative of something with respect to itself), then walk the topological order in reverse, applying the gradient rules above for each operation.

Full Autograd in Action

The interactive playground below puts it all together — graph construction, topological sort, and backward pass. It builds e = tanh(a*b + c), runs backprop, and prints every gradient:

Loading interactive playground...

Try changing the values of a, b, and c and observe how the gradients change. Some things to experiment with:

Set c = 0.0 — now a*b dominates and the tanh is in its steep region
Set a = 0.0 — what happens to de/db?
Set c = 100.0 — the tanh saturates and all gradients become tiny

Understanding the Output

For the default values (a=2, b=-3, c=10):

a*b = -6, so a*b + c = 4
tanh(4) ≈ 0.9993 (nearly saturated)
Because tanh is almost flat at 4, the gradients are very small
de/dc equals the tanh derivative: $1 - \tanh^2(4) \approx 0.0013$
de/da = de/dc * b = 0.0013 * (-3) (negative because b is negative)
de/db = de/dc * a = 0.0013 * 2

Summary

We now have a fully working autograd engine — forward pass, topological sorting, and backward pass. With this machinery, the next logical step is using it to train something: building a loss function, implementing SGD, and watching a model learn.