Say the mini-batch has $N$ samples $(x, y)$,

how will tensorflow utilize this $N$ samples to train the network.

  1. Will it do $N$ forward loop for each sample independently?
  2. Will it do $N$ backward propagation and $N$ weights update for each sample independently?
  3. Or will it average the loss from $N$ samples and then do only 1 weights update?

1 Answer 1

  1. Loss is computed for each sample and then averaged over the entire mini-batch and the weights are updated once. Check this video (start from 6:11) for details.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.