0
$\begingroup$

Why the neural network is updating only after the whole batch passes?

$\endgroup$
1
  • 1
    $\begingroup$ Hi @Goking, we are having trouble answering this question because you haven't provided us with the neural network architecture, implementation and data you are using for this task, as well as context for this issue. Can you please update your post accordingly, so then we can help you? $\endgroup$
    – shepan6
    Commented Jul 7, 2020 at 9:15

1 Answer 1

1
$\begingroup$

Weights get updated based on the number of examples you feed in a batch. This is because, a full pass(forward and backward) of matrix computations has to be completed in order for the weights to be updated, after back-propagation and proceeding with next epoch, with batch type you had chose.

Moreover, If you use stochastic gradient descent, where each example will be processed at a time, your weights will updated after having processed every example.

$\endgroup$
4
  • $\begingroup$ Ok in mini batch Lets say you have batch size of 16. You forward propagate the 16 examples(only changing the values but not weights and biases ) and after the 16th you do a back prop? $\endgroup$
    – Goking
    Commented Jul 7, 2020 at 11:52
  • 1
    $\begingroup$ you forward prop 16 examples till the final layer, then calculate loss, then back prop and then update weights. you have to back prop after every minibatch, since it would not make sense to use mini batch grad descent and not batch grad descent. $\endgroup$ Commented Jul 7, 2020 at 12:00
  • 1
    $\begingroup$ If you want a clear explanation with all the figures and formulaes, watch this video:youtu.be/qzPQ8cEsVK8?list=PLkDaE6sCZn6Ec-XTbcX1uRg2_u4xOEky0 $\endgroup$ Commented Jul 7, 2020 at 12:00
  • $\begingroup$ Also, one of the reasons of using mini batch gradient descent over batch gd is, to make the computational overhead relatively less. So, except for updating parameters every mini batch, everything happens almost the same as batch gd. $\endgroup$ Commented Jul 7, 2020 at 12:39

Not the answer you're looking for? Browse other questions tagged or ask your own question.