I understand that, in a Neural Net, Back Propagation is used to update the model's weights and biases to lower loss, but how does this process actually work?


1 Answer 1


This answer is based on the following series by StatQuest, which I thoroughly recommend.


Back Propagation works by calculating the partial derivatives of the loss function, with respect to each weight and bias in the network, and using those derivatives to alter the value of the corresponding weight or bias.

Considering how the weights and biases are passed through the network, through linear combinations, and Activation Functions, calculating the partial derivatives may seem daunting. However, each partial derivative can be calculated, with reasonable ease, using the Chain Rule from differentiation.

Simple Example

For example, let's use a simple neural network like the one below, which uses ReLU activation functions, and the Squared Sum of Residuals for the loss function.

enter image description here

At the output node, the Squared Sum of Residuals has the following form:

$$\text{Loss} = \sum_{i=0}^{N}(\text{expected}_i - \text{predicted}_i)^2$$

where $i$ represents the $i$th data row to be predicted, $\text{expected}_i$ is the expected value of data row $i$, and $\text{predicted}_i$ is the predicted value of data row $i$. The impact of each parameter (weight or bias) on this loss function can be calculated by by working out its contribution to the loss function. The expected values don't change, so the parameters only come into the function through the predicted values.

We'll use $W_5$ as a simple example, first, let's calculate the partial derivative of the loss function with respect to $W_5$ for a single input value:

\begin{align} \frac{\partial (\text{Loss})}{\partial W_5} &= \frac{\partial ((\text{expected}_0 - \text{predicted}_0)^2)}{\partial W_5} \\ \end{align}

by the chain rule, we have the following:

\begin{align} \frac{\partial ((\text{expected}_0 - \text{predicted}_0)^2)}{\partial W_5} = \frac{\partial ((\text{expected}_0 - \text{predicted}_0)^2)}{\partial \text{predicted}_0} \cdot \frac{\partial (\text{predicted}_0)}{\partial W_5} \end{align}

where $\text{expected}_0$ is the expected value of our single input row, $\text{predicted}_0$ is the predicted value of our single input row, and $W_5$ is our parameter. The first half of the righthand side simplifies to the following:

\begin{align} \frac{\partial ((\text{expected}_0 - \text{predicted}_0)^2)}{\partial \text{predicted}_0} = 2 \cdot (\text{expected}_0 - \text{predicted}_0) \cdot (-1) \end{align}

To get the second half, we first notice that $\text{predicted}_0$ has the following form:

\begin{align} \text{predicted}_0 = W_5 \cdot y_{1, 0} + W_6 \cdot y_{2, 0} + b_3 \end{align}

where $W_5$ and $W_6$ are the weights from the graph, $y_{1, 0}$ and $y_{2, 0}$ are the outputs of hidden layer nodes $1$ and $2$ respectively for the data row $0$, and $b_3$ is the bias applied to the output nodes activation function. Therefore the second half of the righthand side has the following simplified form:

\begin{align} \frac{\partial (\text{predicted}_0)}{\partial W_5} = y_{1, 0} \end{align}


\begin{align} \frac{\partial ((\text{expected}_0 - \text{predicted}_0)^2)}{\partial W_5} = -2 \cdot (\text{expected}_0 - \text{predicted}_0) \cdot y_{1, 0} \end{align}

Multiplying this value by a Learning Rate, for example, 0.001, gives you an update value which you can add to $W_5$ to update the Neural Net and improve its predictive ability.

Deeper Parameters

To extend this process to weights further back in the network you have to perform repeat applications of the chain rule. For example, to get the partial derivative of $\text{Loss}$ with respect to $W_1$ you get the following:

\begin{align} \frac{\partial (\text{Loss})}{\partial W_1} &= \frac{\partial (Loss)}{\partial \text{predicted}_0} \cdot \frac{\partial \text{predicted}_0}{\partial W_1} \\[10pt] &= \frac{\partial (Loss)}{\partial \text{predicted}_0} \cdot \frac{\partial \text{predicted}_0}{\partial y_{1, 0}} \cdot \frac{\partial y_{1, 0}}{\partial W_1} \\[10pt] &= \frac{\partial (Loss)}{\partial \text{predicted}_0} \cdot \frac{\partial \text{predicted}_0}{\partial y_{1, 0}} \cdot \frac{\partial y_{1, 0}}{\partial s_{1, 0}} \cdot \frac{\partial s_{1, 0}}{\partial W_1} \end{align}

where $s_{1, 0}$ is created by summing the linear functions created by multiplying $x_1$ and $x_2$ by their weights $W_1$ and $W_2$ and adding a bias $b_1$, which gives the following form:

\begin{align} s_{1, 0} = W_1 \cdot x_1 + W_2 \cdot x_2 + b_1 \end{align}

The form of the $\frac{\partial y_{1, 0}}{\partial s_{1, 0}}$ term is defined by the activation function $y_{1, 0}$, which in our case is ReLU that gives a gradient of 1 if $s_{1, 0} \gt 0$ and 0 if $s_{1, 0} \leq 0$. If we assume that $s_{1, 0} \gt 0$ we get the following:

\begin{align} \frac{\partial (Loss)}{\partial \text{predicted}_0} \cdot \frac{\partial \text{predicted}_0}{\partial y_{1, 0}} \cdot \frac{\partial y_{1, 0}}{\partial s_{1, 0}} \cdot \frac{\partial s_{1, 0}}{\partial W_1} = -2 \cdot (\text{expected}_0 - \text{predicted}_0) \cdot (W_5) \cdot (1) \cdot (x_1) \end{align}

Wrapping Up

The method above is used to find the gradient for every parameter, then all the parameters are updated to move the Neural Net towards a solution with a lower loss. This process is repeated until, either, you reach an acceptable loss level, or repeat iterations are making negligible improvements to the loss function.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.