I have two models with saved data that worked well previously but won't anymore.

First, it happened with one of my Jupyter notebooks. I can even load the saved model and weights that work. When I train more with the exact same model, the performance actually drops! For example, I get a dice coefficient of -.39 with my previous training when it worked. Now if I load the same model, weights, and data, it drops to -0.04. (Loss of -1 is perfect).

So I load one of my older notebooks with a different model and saved data that worked well. It doesn't converge to nearly as high of a performance as it did previously either.

However, I tried setting up a simple MNIST CNN classifier and it worked fine.

Is there any way for there to be persistent changes to occur so that the exact same code/data that performed well before no longer does?


It was actually just the random weights and incredible luck that I got good convergence the first three tries in a row while didn't most other times afterwards.

I tried setting the seed and found certain seeds that gave me good results every time whereas most other seeds won't converge for many epochs.

from numpy.random import seed
from tensorflow import set_random_seed
  • $\begingroup$ How much data do you have? $\endgroup$ Oct 2 '18 at 9:52
  • $\begingroup$ My consists of simulated "out of focus dots" that overlap with each other. The data is simulated and the pattern is very uniform between the datasets. I don't think I need that many. Alas, I have 1200 samples, each sample is 16 frames of 256x256 images. $\endgroup$
    – yhelothar
    Oct 2 '18 at 11:23
  • $\begingroup$ Then if you have enough data the random split shouldn't be that important... $\endgroup$ Oct 2 '18 at 14:08
  • $\begingroup$ It's the random initial weights that are important. $\endgroup$
    – yhelothar
    Oct 2 '18 at 15:46
  • $\begingroup$ only if you do a very low amount of epoch with big batches, otherwise the weights converge $\endgroup$ Oct 2 '18 at 16:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.