To the best of my understanding , weights are updated during back propagation only using gradient descent and no gradient descent is used during feed forward propagation. is it correct ?
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
The feed-forward pass computes the outputs of each layer, and hence the output of the network, given a current state, i.e. the values for all the parameters for this pass or iteration. The parameters are the weights and biases of the neurons.
Once the feed-forward pass has completed the next step is to update all the parameters in order that in the next feed-forward pass the cost function is smaller. To update the parameters, the parameters are moved in the opposite direction of the gradients of the cost function w.r.t them. This is called Gradient Descent Algorithm, and basically, taking into account that the gradients indicates the direction of steepest increase of the cost function, if the parameters are moved towards the opposite direction, the next step the cost function is supposed to be smaller.
The problem with neural networks is that we don't know the gradients of the cost function because the number of the parameters are massive (billions) and the cost function themselves are quite complex (not convex, with pathological curvatures...) To overpass this issue, backpropagation has invented as a means of efficiently compute the gradients of the cost function (it uses the chain rule and propagates the errors from the lat layer to the first one)
To the best of my understanding , weights are updated during back propagation only using gradient descent and no gradient descent is used during feed forward propagation. is it correct ?
Yes and no...
- Yes if you think of gradient descent as only the updating stage of the parameters
- No if you understand Gradient Descent as a whole, in which you need the results obtained in the forward pass to compute the values of the gradients in the backwards pass
I'm personally prone to the second option, in which Gradient Descent comprises two stages: a gradient computation and parameter updating
