0
$\begingroup$

In the context of Machine Learning, I encounter often the fact that a correction step does not occur after each training step, but only every n learning steps.

Citing from the Deep Learning with Python book:

This is the training loop, which, repeated a sufficient number of times (typically tens of iterations over thousands of examples), yields weight values that minimize the loss function

Why don't we correct at every step, but typically only once every 100 learning samples?

I assume, but I am not sure, that this might be because of efficiency, and also to smooth the correction "path" (e.g. integrating a correction step that is an average of the last 100 loss function values).

Thank you in advance!

$\endgroup$
1
$\begingroup$

This is called batch updation which is the most popular method of updating weights. We also call it BatchSGD in context of SGD. Yes, what you mentioned is true. If we have 1000's of weights which we typically have in deep neural networks. It is not efficient to calculate partial derviatives at each weights for every input. Instead, We do the batch updation by which we will aggregate all the loss for last 100 inputs(as in your case) and at the end of 100th input, we take the average fo the losses and update the weight of the network. Keep in mind that calculation partial derivatives is one of the most compute intensive tasks that we perform in neural networks. So making 100 updates to Just 1 save a lot of compute. Hope it helps

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.