# How to assess that a cross entropy based model has converged

I have a question regarding cross entropy convergence using Stochastic Gradient Descent. I am a little bit confused about how the convergence should be assessed. Should we treat the model as converged if the loss hit minimum on any single example or may be a certain number of examples? I am asking because what if the model converged on a single example just randomly.

Pure, hardcore Stochastic Gradient Descent (when you feed just one observation at a time) is not advisable at all. The descent of the Gradient is so noisy that after a certain minimal loss reduction it will stop learning anything. It will wander around the loss function in unpredictable ways. In this case, you're right: there's no way to assess final convergence. Even if the Gradient hits the global minimum, it will jump away from it at the following iteration.

The most powerful technique is a robust version of SGD: Mini-Batch Gradient Descent. In this case, you feed a batch of data at each training iteration (usually of size between 30 to 250, but it's up to you). Batch size is another hyperparameter, therefore you must find a trade off between robustness to noise on one side, and on the other side to avoid get stuck in local minima and lower computational times.

• PS: On this respect, I don't think Cross-Entropy makes a difference with respect to other Loss functions, for this reason I think these conclusions are also valid for other Losses. Dec 30 '19 at 11:45
• Good explanation, thanks
– Jim
Dec 30 '19 at 14:19

When a loss of your model on a subset of examples is calculated you are trying to estimate the "true" loss of your model on the underlying distribution of training examples. The loss of a single training example is a bad estimate for the expected loss of your model on the whole population for the exact reason you are mentioning: it might be right or wrong by chance. (=the estimate of the loss has very high variance.)

Thus, any reasonable model fitting procedure estimates the training loss by taking the mean of the loss over a subset of examples or even the whole training set. This will be a good estimate for you to judge if the model training converged.

• thanks, it clarifies much for me
– Jim
Dec 30 '19 at 14:20