Backpropagation Explained for Beginners (Part 2): There Has to Be a Better Way

Welcome back! Let's continue our journey of understanding backpropagation in detail.

Let's briefly recall what we covered in Part 1 so far.

A Quick Recap

We first started trying to understand neural networks using a simple dataset.

In that process, we built a small neural network and understood how it makes predictions through forward propagation.

We then observed that the predicted values were far from the actual values, resulting in a large error. Now, we wanted to train this neural network to perform better.

By comparing it with simple linear regression, we observed that, in our neural network, the loss depends on seven parameters: $w\_1,\; w\_2,\; w\_3,\; w\_4,\; b\_1,\; b\_2,\; b\_3$.

Our complete loss function looked like this:

$$L(w\_1,w\_2,w\_3,w\_4,b\_1,b\_2,b\_3)\; =\; \backslash frac\{1\}\{n\}\; \backslash sum\_\{i=1\}^\{n\}\; \backslash left(\; y\_i\; -\; \backslash left(\; w\_3\; \backslash text\{ReLU\}(w\_1\; x\_i\; +\; b\_1)\; +\; w\_4\; \backslash text\{ReLU\}(w\_2\; x\_i\; +\; b\_2)\; +\; b\_3\; \backslash right)\; \backslash right)^2$$

Next, just as we did in simple linear regression, we wanted to differentiate the loss with respect to each parameter. We started with $w\_1$ and used the classical differentiation method, arriving at:

$$\backslash frac\{\backslash partial\; L\}\{\backslash partial\; w\_1\}\; =\; -\backslash frac\{2\}\{n\}\; \backslash sum\_\{i=1\}^\{n\}\; (y\_i\; -\; \backslash hat\{y\}\_i)\; \backslash ,\; w\_3\; \backslash ,\; \backslash text\{ReLU\}\text{'}(w\_1\; x\_i\; +\; b\_1)\; \backslash ,\; x\_i$$

Do We Really Have to Repeat This?

Now it's time to think. We differentiated the loss with respect to $w\_1$, meaning we were finding how the loss changes as $w\_1$ changes. We used the classical differentiation method. However, we have six more parameters to differentiate with respect to. Do we need to proceed with the same method? No.

The Chain Rule to the Rescue

If you remember, we discussed the chain rule in Part 1. We used a simple example: $y\; =\; x^2$ and $z\; =\; y^3$. We noticed that $z$ does not depend directly on $x$. Instead, the relationship looks like $x\; \backslash rightarrow\; y\; \backslash rightarrow\; z$. This means that when $x$ changes, it first changes $y$, and the change in $y$ then affects $z$. Since the chain rule efficiently captures such dependencies, we can apply it to our neural network to compute gradients without re-deriving each derivative from scratch. In the context of modern deep learning (2026), frameworks like TensorFlow and PyTorch automate this process using automatic differentiation, but understanding the chain rule remains fundamental for debugging and optimizing models.

via Towards Data Science