1
$\begingroup$

I am confused that in ANN, we use Gradient Descent to find the curve of the optimal points of predicted values.

Let's say we use MSE (Mean Squared Error) and the lost function is 1/2m * (y-y')^2

And the cost function is sum of lost function values. What I've studied is that we get an equation after the cost function and we get the derivative of this equation for our linear line to get the Gradient points to calculate the slope and intercept.

But now I am watching a video and searched through the internet, I've come across that people use their activation function for back-propogation.

For instance in the video I've watched, he uses the derivative of the Sigmoid Function for back-propogation.

Which one is correct? I am really confused.

$\endgroup$

1 Answer 1

2
$\begingroup$

Doing back propagation is a step-wise procedure (you propagate back one step after another), which - if you do the math - comes from the chain rule. So basically, you have to do both the derivative of your costs as well as the derivative of the activation function.

Example

Here is a small example. We start with a small network:
enter image description here

Mathematical setting

Following your notion, the output of the network is $y'=f_B(z_B)$ where $f_B$ is the activation function of node $B$ and $z_B$ is the input of node $B$.
The input $z_B$ is then given by the weighted sum of the outputs from both nodes $A_1$ and $A_2$. Let's call the outputs $o_1,o_2$ and the weights $v_1$ and $v_2$, i.e. $$z_B = v_1\cdot o_1 + v_2\cdot o_2$$ The rest of the network is similar (e.g. $o_1=f_{A_1}(z_{A_1})$).

Our loss for a single sample $i$ would then be $\mathcal{L}_i = \frac{1}{2}(y-y')^2$. And the costs would be the average loss $\mathcal{C} = \frac{1}{m}\sum_{i=1}^m \mathcal{L_i}$.
Note Formally, I would have to add the sample index $i$ also to $y'$, $z_B$, $o_1$, $o_2$, since they differ for each sample. I omit it here for the sake of a simple notion.

Backpropagation

The goal is to find weights $v_1$ and $v_2$ that minimize the costs.

Step 1: We start - as you heard before - with the derivative of the costs $$\frac{\partial}{\partial v_1}\mathcal{C}=\frac{1}{m}\sum_{i=1}^m\frac{\partial}{\partial v_1}\mathcal{L}_i$$

Step 2: We compute the derivative of the sample loss. This is the point where the chain rule is used: $$\frac{\partial}{\partial v_1}\mathcal{L}_i = (y-y')\cdot \frac{\partial}{\partial v_1}y'$$

Step 3: We need to compute derivative the output $y'=f_B(z_B)$ in order to compute Step 2. Again, we use the chain rule, to propagate the derivative out the cost further back: $$\frac{\partial}{\partial v_1}y' = f_B'(z_B)\cdot \frac{\partial}{\partial v_1}z_B$$ Here, $f_B'$ is the derivative of the activation function $f_B$

Step 4: We need to compute the derivative of $z_B$ in order to compute Step 3. This is simple: $$\frac{\partial}{\partial v_1}z_B = \frac{\partial}{\partial v_1}(v_1\cdot o_1 + v_2\cdot o_2) = o_1$$

If we now put everything together, we finally get the derivative of the costs. So the derivative of the activation function is needed to achieve the bigger goal of computing the derivative of the costs.

Further steps

You would also want to optimize the lower layers of the network, e.g. $z_{A_1}=w_{11}x_1 + w_{12}x_2$, so you would also compute $\frac{\partial}{\partial w_{11}}\mathcal{C}$, which would start in the same way and differ in step 4 and require more steps to propagate the derivative further back. Luckily, the backpropagation for all weights can be done at the same time, which leads to the backpropagation algorithms, you learn about.

$\endgroup$
1
  • $\begingroup$ Thank you so much!! $\endgroup$
    – Mystheman
    Jan 3, 2023 at 9:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.