I have two models with saved data that worked well previously but won't anymore.
First, it happened with one of my Jupyter notebooks. I can even load the saved model and weights that work. When I train more with the exact same model, the performance actually drops! For example, I get a dice coefficient of -.39 with my previous training when it worked. Now if I load the same model, weights, and data, it drops to -0.04. (Loss of -1 is perfect).
So I load one of my older notebooks with a different model and saved data that worked well. It doesn't converge to nearly as high of a performance as it did previously either.
However, I tried setting up a simple MNIST CNN classifier and it worked fine.
Is there any way for there to be persistent changes to occur so that the exact same code/data that performed well before no longer does?