Earlystopping in multi-output deep learning

Question

When working with a neural network with more than one output, what is generally advised as the best strategy for early-stopping the training process?

Given that I am currently monitoring the net validation loss (validation loss from n different output neurons added together), and have noticed my training early-stopping before convergence in one or more of my losses, I am curious as to what the best practices in this situation are.

• Should one monitor the net validation loss, or is there any way to implement early-stopping such that m out of n (m goes from 1 to n) outputs' validation losses are monitored and training stops at convergence of all/oscillations within a certain threshold.
• How would one implement this without risking overtraining some output nodes while preventing others from under-training?

I am a newcomer to this field, if you hadn't guessed already so please bear with me if this question is not extremely well put together!

Answer 1

How would one implement this without risking overtraining some output nodes while preventing others from under-training?

This isn't done with early stopping. There are rather other techniques that could prevent that some main nodes are trained in your model such as Dropout. By using Dropout your model can't rely on some main nodes, because they could get shut off at any time. So with Dropout the model should learn to balance the weights throughout the nodes.

Should one monitor the net validation loss, or is there any way to implement early-stopping such that m out of n (m goes from 1 to n) outputs' validation losses are monitored and training stops at convergence of all/oscillations within a certain threshold.

You noted that n in your case refers to the number of output nodes. Generally, that is the number of classes in your classifier which is mostly denoted by c. So, in multi-class classification you wouldn't have c different validation losses but average them to obtain one mean validation loss for the whole model.

How would one implement ...

There is no gold rule to how to implement early stopping. The simplest way to do it is as follows:

• Set a so called patience i.e. after how many epochs do we stop if the loss doesn't improve (usually set to 10)
• After each epoch check your validation loss
• Then select the model patience epochs before you stopped, because that was the best performing model.

You see that this is not the best approach and there could be better ones. In this paper, some of them are suggested.

... training stops all/oscillations within a certain threshold

Could be another idea, that would have to be checked. I wouldn't recommend it though, because with SGD you still have oscillations at the very beginning.

Answer 2

I know this is a 4-year old question but anyway.
If you are using Keras there's a parameter called loss_weights= in the model compile method. So if you have output_1 and output_2 and output_1 converges first you can try loss_weights=[0.2, 0.8]. IIRC the default loss in Keras is the sum of loss for outputs, not average, so here I've also normalized the loss so it is comparable across models with variable numbers of outputs (e.g. if you want to use standard values for learning rates but those are for models with single outputs and you have a hundred outputs).