Welcome back! First, I want to extend a huge thank you for the incredible response to the first two parts of this series. Knowing that so many of you have found them helpful means a lot.
As we dive into Part 3, I encourage you to share any thoughts, questions, or suggestions as you read—I genuinely appreciate your perspective.
Why Recompute the Same Gradients?
In Part 2, we used the chain rule to calculate the gradient for \(w_1\):
\[ \frac{\partial L}{\partial w_1} = \frac{\partial L}{\partial \hat{y}} \cdot \frac{\partial \hat{y}}{\partial a_1} \cdot \frac{\partial a_1}{\partial z_1} \cdot \frac{\partial z_1}{\partial w_1} \]This gave us the same equation we had derived using classical differentiation in Part 1:
\[ \frac{\partial L}{\partial w_1} = -2(y-\hat{y}) \cdot w_3 \cdot \mathrm{ReLU}'(w_1x+b_1) \cdot x \]This highlighted how powerful the chain rule is. Now, what about the gradients for the other parameters: \(b_1, w_2, b_2, w_3, w_4, b_3\)?
Intuitively, we could repeat the chain rule process for each parameter. But if we write out the full set of equations, we see a pattern: many of the same partial derivatives appear repeatedly across different parameter gradients.
\[ \begin{align*} \frac{\partial L}{\partial w_1} &= \frac{\partial L}{\partial \hat{y}} \cdot \frac{\partial \hat{y}}{\partial a_1} \cdot \frac{\partial a_1}{\partial z_1} \cdot \frac{\partial z_1}{\partial w_1} \\ \frac{\partial L}{\partial b_1} &= \frac{\partial L}{\partial \hat{y}} \cdot \frac{\partial \hat{y}}{\partial a_1} \cdot \frac{\partial a_1}{\partial z_1} \cdot \frac{\partial z_1}{\partial b_1} \\ \frac{\partial L}{\partial w_2} &= \frac{\partial L}{\partial \hat{y}} \cdot \frac{\partial \hat{y}}{\partial a_2} \cdot \frac{\partial a_2}{\partial z_2} \cdot \frac{\partial z_2}{\partial w_2} \\ \frac{\partial L}{\partial b_2} &= \frac{\partial L}{\partial \hat{y}} \cdot \frac{\partial \hat{y}}{\partial a_2} \cdot \frac{\partial a_2}{\partial z_2} \cdot \frac{\partial z_2}{\partial b_2} \\ \frac{\partial L}{\partial w_3} &= \frac{\partial L}{\partial \hat{y}} \cdot \frac{\partial \hat{y}}{\partial w_3} \\ \frac{\partial L}{\partial w_4} &= \frac{\partial L}{\partial \hat{y}} \cdot \frac{\partial \hat{y}}{\partial w_4} \\ \frac{\partial L}{\partial b_3} &= \frac{\partial L}{\partial \hat{y}} \cdot \frac{\partial \hat{y}}{\partial b_3} \end{align*} \]Take \(b_1\) for instance. Its chain rule equation is:
\[ \frac{\partial L}{\partial b_1} = \frac{\partial L}{\partial \hat{y}} \cdot \frac{\partial \hat{y}}{\partial a_1} \cdot \frac{\partial a_1}{\partial z_1} \cdot \frac{\partial z_1}{\partial b_1} \]Compare this to \(w_1\)'s equation—notice that the first three terms are identical:
\[ \frac{\partial L}{\partial \hat{y}}, \quad \frac{\partial \hat{y}}{\partial a_1}, \quad \frac{\partial a_1}{\partial z_1} \]In fact, for any parameter in the same layer (or path), the initial partial derivatives from the loss back to that layer are the same. The only difference is the final factor: \(\frac{\partial z_1}{\partial w_1}\) versus \(\frac{\partial z_1}{\partial b_1}\).
This duplication is a clue: instead of computing each gradient from scratch, we can compute the shared 'error signal' once and reuse it. That is the core idea behind backpropagation.
The Backpropagation Algorithm: A Smarter Approach
Rather than repeating the chain rule for each parameter, backpropagation works in two passes:
- Forward pass: Compute the network's output (and loss) given the current weights.
- Backward pass: Propagate the error backward through the network, computing the gradient of the loss with respect to each parameter, but reusing intermediate results (the 'local gradients') as much as possible.
This is essentially applying the chain rule in a structured way, layer by layer, from output to input. The shared partial derivatives are computed once and used for all parameters in that layer, dramatically reducing redundant computation.
In practice, backpropagation is implemented in deep learning frameworks (e.g., PyTorch, TensorFlow) via automatic differentiation. As of 2026, these frameworks have made it effortless to train complex models, but understanding the underlying mechanics remains key for debugging and architecture design.
What's Next?
Now that we see why recomputing gradients is wasteful, in the next part we'll formalize the backpropagation algorithm step by step. We'll introduce the concept of the 'error term' \(\delta\) and show how to propagate it backward efficiently.
Stay tuned, and feel free to ask questions in the comments below!