0
$\begingroup$

I have a very basic question on the optimization algotithm, when I'm adjusting weights and biases in a NN, should I:

  1. Forward propagate and backpropagate to calculate gradient descent (DC) for each batch once and then repeat for iterations_number times.

or

  1. Forward propagate and backpropagate to calculate gradient descent (DC) one batch for iterations_number times and then continue with the next batch.
$\endgroup$
0
$\begingroup$

One iteration means that you do one forward pass and one backward pass for a single batch containing batch_size examples. Then you move on to the next batch. Also see this post on SE SO.

Note that this is not identical to an epoch. An epoch is only completed when all examples of your dataset have been passed through your network. And as long as your batch size is less than the number of datapoints you will need $\frac{n}{\text{batch size}}$ (with $n$ being the sample size) iterations to complete an epoch. Also see this answer on SE DS.

$\endgroup$
2
  • $\begingroup$ I see, so answer is option 1. and the correct terminology should be epochs_number instead of iterations_number as the second is just $\frac{n}{batch size}$. $\endgroup$ Jan 16 at 18:13
  • $\begingroup$ @OliverMohrBonometti Yes, then it would be correct. The number of iterations is then calculated as following: iterations_number = (n / batchsize) * epochs_number. $\endgroup$
    – Sammy
    Jan 17 at 8:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.