23
$\begingroup$

Please forgive me as I am new to this. I have attached a diagram trying to model my understanding of neural network and Back-propagation? From videos on Coursera and resources online I formed the following understanding of how neural network works:

  1. Input is given, which gets weight assigned to it using a probability distribution.
  2. The activation functions use the weights to provide the predicted value.
  3. The cost or loss functions calculate the error of the prediction between the actual class and the predicted value.
  4. The optimization functions such as gradient descent use the results of the cost function to minimize the error.

If the above is correct then I am struggling to understand the connection between Gradient Descent and Backpropagation?

Here is an image of my understanding so far:

Improve this question
$\endgroup$

1 Answer 1

Reset to default
20
$\begingroup$

First, remember that the derivative of a function gives the direction in which the function increases, and its negative, the direction in which the function decreases.

Training a model is just minimising the loss function, and to minimise you want to move in the negative direction of the derivative. Back-propagation is the process of calculating the derivatives and gradient descent is the process of descending through the gradient, i.e. adjusting the parameters of the model to go down through the loss function.

Back-propagation is called like this because to calculate the derivative you use the chain rule from the last layer (which is the one directly connected to the loss function, as it is the one that provides the prediction) to the first layer, which is the one that takes the input data. You are "moving from back to front".

In gradient descent one is trying to reach the minimum of the loss function with respect to the parameters using the derivatives calculated in the back-propagation. The easiest way would be to adjust the parameters by substracting its corresponding derivative multiplied by a learning rate, which regulates how much you want to move in the gradient direction. But there are some more advanced algorithms like for example ADAM.

Improve this answer
$\endgroup$
4
  • 1
    $\begingroup$ Thank you for your time and clearing things up. For those who are new to ML, I went back to review a video on coursera from Prof Andrew Ng and it helped me a lot too. $\endgroup$
    – Mohamed Mahyoub
    Jan 30, 2019 at 9:27
  • $\begingroup$ I am glad it helped. $\endgroup$
    – Escachator
    Jan 30, 2019 at 21:59
  • $\begingroup$ Thanks a lot! Do you know a scheme where this is visualized simply? Means, how the chain rule is applied to the certain weights and so? $\endgroup$
    – Ben
    Jun 4, 2021 at 12:49
  • $\begingroup$ Clear explanation, however, backpropagation calculates the errors(delta) from the last layer(output), till the second one L2(the first hidden layer if any), and not the input layer as you mentioned. $\endgroup$
    – Zeinab
    Nov 27, 2022 at 12:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.