Building an Autograd Engine

Recap

In Chapter 1, we learned that deep learning is function composition + the chain rule. Now we'll build the data structures that make autograd work.

---

The Core Data Model

Our autograd engine needs three things:

1. Value — a node in the graph that stores a number (data), its gradient (grad), and how it was produced (op)
2. Op — a tagged union describing the operation: Add(a, b), Mul(a, b), Tanh(a), or None for leaf nodes
3. Graph — an array of Values, where nodes refer to each other by index

Why indices instead of pointers? It makes the graph easy to copy, avoids shared mutation.

Each Op variant stores the indices of its input nodes. None means the node is a leaf (e.g., a parameter or input value).

Graph Construction

Every operation pushes a new Value into graph.nodes and returns its index. This is how the graph grows:

• MakeValue(g, x) creates a leaf node with zero gradient and Op::None
• VAdd(g, a, b) computes a.data + b.data and stores Op::Add(a, b)
• VMul(g, a, b) computes a.data * b.data and stores Op::Mul(a, b)
• VTanh(g, a) computes tanh(a.data) and stores Op::Tanh(a)

Each op does two things: forward compute (calculate the data value) and record (store the Op with operand indices). Think of it as: "compute now, remember how to differentiate later."

This is exactly the "tape" that autograd libraries record. The difference is we're doing it explicitly with a simple array.

Let's Build a Graph

The interactive playground below defines all the data structures and builds the graph for e = tanh(a * b + c). Read through the code, then press ▶ Run to inspect every node:

Loading interactive playground...

Try changing a, b, or c and see how the final result changes. Notice that the graph has 6 nodes: three leaves (a, b, c), the product (a*b), the sum (a*b+c), and the final tanh.

Summary

We now have a computation graph that records operations and computes forward values. Every gradient is still zero — the next step is implementing topological sorting and the backward pass to fill in those gradients using the chain rule.