3
$\begingroup$

Please forgive me as I am new to this. I have attached a diagram trying to model my understanding of neural network and Back-propagation? From videos on coursera and resources online I formed the following understanding of how neural network works:

  1. Input is given which get's weight assigned to it using a probability distribution
  2. The activation functions use the weights to provide the predicted value
  3. The cost or loss functions calculate the error of the prediction between the actual class and the predicted value
  4. The optimization functions such as gradient decent use the results of the cost function to minimize the error.

If the above is correct then I am struggling to understand the connection between Gradient Descent and Backpropagation ?

Here is an image of my understanding so far:

Many thanks in advance.

$\endgroup$
3
$\begingroup$

First, remember that the derivative of a function gives the direction in which the function increases, and its negative, the direction in which the function decreases.

Training a model is just minimising the loss function, and to minimise you want to move in the negative direction of the derivative. Back-propagation is the process of calculating the derivatives and gradient descent is the process of descending through the gradient, i.e. adjusting the parameters of the model to go down through the loss function.

Back-propagation is called like this because to calculate the derivative you use the chain rule from the last layer (which is the one directly connected to the loss function, as it is the one that provides the prediction) to the first layer, which is the one that takes the input data. You are "moving from back to front".

In gradient descent one is trying to reach the minimum of the loss function with respect to the parameters using the derivatives calculated in the back-propagation. The easiest way would be to adjust the parameters by substracting its corresponding derivative multiplied by a learning rate, which regulates how much you want to move in the gradient direction. But there are some more advanced algorithms like for example ADAM.

$\endgroup$
  • 1
    $\begingroup$ Thank you for your time and clearing things up. For those who are new to ML, I went back to review a video on coursera from Prof Andrew Ng and it helped me a lot too. $\endgroup$ – Mohamed Mahyoub Jan 30 '19 at 9:27
  • $\begingroup$ I am glad it helped. $\endgroup$ – Escachator Jan 30 '19 at 21:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.