4
$\begingroup$

I have a training procedure set up for an image recognition task. Each time I train a model, I record training loss, validation loss, validation precision and validation recall.

Recently I switched from EfficientNet to a ResNet-based model. Both models are pretrained, so weight initialization is deterministic. With the old model I ran 5 experiments (each on 5-folds) with exactly the same parameters, varying only the seed and got around 0.001 standard deviation for validation loss. With the new model I ran 3 experiments (also with 5 folds) with exactly the same parameters and got a standard deviation of 0.028 for validation loss.

This high variance makes it very hard to interpret the effects of changing non-seed parameters when running new experiments with the new model, since in many cases the differences in performance sit within one standard deviation. Experiments take multiple hours each, so it's not feasible to run multiple experiments for each condition.

What does one do in cases like this?

$\endgroup$
6
  • $\begingroup$ The second model is not stable, or at least not as stable as the first one... it's a bad sign, unless the second model has a big advantage over the first one in average performance. $\endgroup$
    – Erwan
    Nov 27 '20 at 2:04
  • $\begingroup$ Yeah, it certainly is not stable and I agree it's a bad sign, but it does appear to have a big advantage. Are there any resources that look into why models are unstable or how to make them more stable? I'm having a tough time dredging up anything useful. $\endgroup$ Nov 27 '20 at 5:46
  • $\begingroup$ I don't know this kind of model so I can't tell you. In general this happens when the model is overfit, which is usually because there's not enough training data and/or too many parameters to estimate. Hopefully somebody else will have a more specific answer. $\endgroup$
    – Erwan
    Nov 27 '20 at 11:44
  • $\begingroup$ Thanks @Erwan, that's really useful to know! In this case the training and validation loss are very similar, so I don't think it's overfitting, but the second model IS larger than the first, so maybe there is not enough training data for the second model. Do you happen to have a source for that insight? I'd love to read more if so. $\endgroup$ Nov 27 '20 at 23:31
  • $\begingroup$ You're right, the fact that the training and validation loss are close makes overfitting unlikely. Then I wonder if it could be underfitting maybe? That doesn't seem likely either since the model is more complex, but it could happen if the model architecture doesn't fit the data (this an hypothesis, I have no idea if it's correct). Sometimes it's really difficult to know with real world models. About sources: the point I mentioned is a consequence of the bias-variance tradeoff, you would find many questions related to it ... $\endgroup$
    – Erwan
    Nov 28 '20 at 0:47
0
$\begingroup$

It's just a simple idea that I often recommend to understand what happens in cases like this:

You could try to do an ablation study, where you train the model with varying sizes of training data (e.g. 10%, 20%,..., 100% of your full training data). Given that your problem is instability, it would be even better to do every size several times with different random subsets. Then you plot the performance as a function of the data size, for example with boxplots in order to visualize the variance. The evolution of the performance (and its variance) would show you whether there is enough training data given the complexity of the model: if yes, the curve increases and then starts stabilizing around its maximum. If not, the curve is still in a phase of increase when it reaches 100%, meaning that more data would be needed to allow the model to stabilize.

Of course the disadvantage is that you need to run the training/testing many times, so this might not be practical.

$\endgroup$
0
$\begingroup$

Ok so not a full answer, but I will mention that in my case simply training for more epochs reduced the model variance:

enter image description here

As you can see, training for 9 epochs results in lower and more consistent val loss results. Things are still a bit unworkable since a single 9 epoch experiment takes > 6 hours, but it's something.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.