# Models that converged before aren't converging anymore in Keras

I have two models with saved data that worked well previously but won't anymore.

First, it happened with one of my Jupyter notebooks. I can even load the saved model and weights that work. When I train more with the exact same model, the performance actually drops! For example, I get a dice coefficient of -.39 with my previous training when it worked. Now if I load the same model, weights, and data, it drops to -0.04. (Loss of -1 is perfect).

So I load one of my older notebooks with a different model and saved data that worked well. It doesn't converge to nearly as high of a performance as it did previously either.

However, I tried setting up a simple MNIST CNN classifier and it worked fine.

Is there any way for there to be persistent changes to occur so that the exact same code/data that performed well before no longer does?

It was actually just the random weights and incredible luck that I got good convergence the first three tries in a row while didn't most other times afterwards.

I tried setting the seed and found certain seeds that gave me good results every time whereas most other seeds won't converge for many epochs.

from numpy.random import seed
seed(5)
from tensorflow import set_random_seed
set_random_seed(42)

• How much data do you have? Oct 2 '18 at 9:52
• My consists of simulated "out of focus dots" that overlap with each other. The data is simulated and the pattern is very uniform between the datasets. I don't think I need that many. Alas, I have 1200 samples, each sample is 16 frames of 256x256 images. Oct 2 '18 at 11:23
• Then if you have enough data the random split shouldn't be that important... Oct 2 '18 at 14:08
• It's the random initial weights that are important. Oct 2 '18 at 15:46
• only if you do a very low amount of epoch with big batches, otherwise the weights converge Oct 2 '18 at 16:08