0
$\begingroup$

I have a question regarding cross entropy convergence using Stochastic Gradient Descent. I am a little bit confused about how the convergence should be assessed. Should we treat the model as converged if the loss hit minimum on any single example or may be a certain number of examples? I am asking because what if the model converged on a single example just randomly.

$\endgroup$
1
$\begingroup$

Pure, hardcore Stochastic Gradient Descent (when you feed just one observation at a time) is not advisable at all. The descent of the Gradient is so noisy that after a certain minimal loss reduction it will stop learning anything. It will wander around the loss function in unpredictable ways. In this case, you're right: there's no way to assess final convergence. Even if the Gradient hits the global minimum, it will jump away from it at the following iteration.

The most powerful technique is a robust version of SGD: Mini-Batch Gradient Descent. In this case, you feed a batch of data at each training iteration (usually of size between 30 to 250, but it's up to you). Batch size is another hyperparameter, therefore you must find a trade off between robustness to noise on one side, and on the other side to avoid get stuck in local minima and lower computational times.

$\endgroup$
2
  • $\begingroup$ PS: On this respect, I don't think Cross-Entropy makes a difference with respect to other Loss functions, for this reason I think these conclusions are also valid for other Losses. $\endgroup$
    – Leevo
    Dec 30 '19 at 11:45
  • $\begingroup$ Good explanation, thanks $\endgroup$
    – Jim
    Dec 30 '19 at 14:19
1
$\begingroup$

When a loss of your model on a subset of examples is calculated you are trying to estimate the "true" loss of your model on the underlying distribution of training examples. The loss of a single training example is a bad estimate for the expected loss of your model on the whole population for the exact reason you are mentioning: it might be right or wrong by chance. (=the estimate of the loss has very high variance.)

Thus, any reasonable model fitting procedure estimates the training loss by taking the mean of the loss over a subset of examples or even the whole training set. This will be a good estimate for you to judge if the model training converged.

$\endgroup$
1
  • $\begingroup$ thanks, it clarifies much for me $\endgroup$
    – Jim
    Dec 30 '19 at 14:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.