Training a Neuron

Recap

In Chapter 3, we built a complete autograd engine: forward pass, topological sort, backward pass. Now we'll use it to train something.

---

A Single Neuron

The simplest neural network is one neuron. It computes:

$\hat{y} = \tanh(w \cdot x + b)$

where:

• $x$ is the input (given)
• $w$ is the weight (learned)
• $b$ is the bias (learned)
• $\hat{y}$ is the prediction

The tanh is the activation function. Without it, stacking multiple layers would collapse into a single linear transformation. Non-linearity is what makes deep learning deep.

The Loss Function

To train, we need a loss — a single number that tells us how wrong the model is. For a single data point with target $y$:

$L = (\hat{y} - y)^2$

This is the mean squared error (MSE). When $\hat{y} = y$, the loss is zero. Otherwise, it's positive.

Stochastic Gradient Descent (SGD)

Training is a loop:

1. Forward: compute $\hat{y}$ and $L$ (builds the graph)
2. Backward: call Backward(graph, loss) to fill in gradients
3. Update: nudge each parameter opposite to its gradient

$\theta \leftarrow \theta - \eta \cdot \frac{\partial L}{\partial \theta}$

where $\eta$ is the learning rate. Too high → diverges. Too low → painfully slow.

4. Reset: create a fresh graph (or zero out gradients) and repeat

Training a Neuron — Live!

This interactive demo trains a single neuron to learn y = tanh(2*x + 1) for a few data points. Watch the loss decrease as the neuron figures out the right weight and bias:

Loading interactive playground...

Experiments to Try

• Change the learning rate: try lr = 0.1 (slower) or lr = 2.0 (might diverge!)
• Change the epochs: try epochs = 50 for more training
• Change initial weights: try w_val = -1.0 — can it still converge?
• Add more data points: add 3.0 to xs and the corresponding target to ys

Common Gotchas

1. Forgetting to reset the graph: If you reuse a graph, old gradients accumulate. We create a fresh Graph each epoch to avoid this.

2. Missing non-linearity: Without tanh, stacking multiple linear layers is equivalent to a single linear layer. Non-linearities are what give neural networks their expressive power.

3. Exploding/vanishing gradients: Very deep graphs can make gradients too large (exploding) or too small (vanishing). tanh is particularly prone to vanishing gradients when saturated. This is why modern networks often use ReLU.

4. Learning rate sensitivity: SGD is sensitive to the learning rate. Adaptive optimizers like Adam adjust the learning rate per-parameter, but SGD is the foundation they all build on.