# Why does my minimal CNN example show strongly fluctuating validation loss?

I'm fairly new at working with neural networks and think I am making some basic mistake. I am trying to assign simulated images to 5 classes to test, what (if any) networks are helpful for a large data problem we have in our group. I am training the ResNet50 CNN included in Keras on 40000 (256 by 256 pixel greyscale) images. While training loss is improving quickly within the first epoch, validation loss fluctuates wildly in (to me) fairly random manner.

I am trying to use as many high level functions as possible and have ended up with the following code:

from keras.preprocessing.image import ImageDataGenerator
from keras.applications.resnet50 import ResNet50
from keras.models import Model
from keras.layers import GlobalAveragePooling2D, Dense

inputShape = (256, 256,1)
targetSize = (256, 256  )
batchSize = 128

base_model = ResNet50(include_top =False,
weights =None,
input_shape=inputShape)
x = base_model.output

# modified based on the article https://github.com/priya-dwivedi/
# Deep-Learning/blob/master/resnet_keras/Residual_Networks_yourself.ipynb

x = GlobalAveragePooling2D()(x)
predictions = Dense(5, activation= 'softmax')(x)
model = Model(inputs = base_model.input, outputs = predictions)

# Compile model, might want to change loss and metrics
model.compile(loss='categorical_crossentropy',
metrics = ["acc"]
)

# Define the generators to read the data. Training generator performs data argumentation as well
train_datagen = ImageDataGenerator(
samplewise_center =True,
samplewise_std_normalization = True,
shear_range=0.2,
zoom_range=0.2,
horizontal_flip=True)

test_datagen = ImageDataGenerator(
samplewise_center =True,
samplewise_std_normalization = True,
)

train_generator = train_datagen.flow_from_directory(
'trainingData/FolderBasedTrain256',
target_size=targetSize,
color_mode = 'grayscale',
batch_size=batchSize,
save_to_dir = 'trainingData/argumentedImages',
class_mode='categorical')

validation_generator = test_datagen.flow_from_directory(
'trainingData/FolderBasedValidation256',
target_size=targetSize,
batch_size=batchSize,
color_mode = 'grayscale',
class_mode='categorical')

# Fit the Modell, saving the history for later plotting
history = model.fit_generator(
train_generator,
epochs=20,
validation_data=validation_generator,
#steps_per_epoch = 62,
steps_per_epoch = 31,
#callbacks=[checkpointer],
validation_steps=9)


I can always create more images or train longer, but to me, this looks as if something fundamental went wrong somewhere. I would be very gratefull for any and all ideas. Thank you!

EDIT: I was told to stress that validation and training set are both created by exaclty the same simulation routine, so they should be relativly easy to classify

EDIT2: Found the error! My batch size did not fit with the amount of data and the steps by epoch, resulting in the CNN not seeing all the training data. Now everything converges nicely and I can evaluate the choice of modell. Thanks to all contributers

• Are you computing validation metrics on only batchSize examples? Maybe you should use more examples to get a more robust estimate of those metrics? – kbrose Mar 21 '19 at 13:20
• As far I understand, "validation_steps=9" in model.fit_generator should make it so that the entire validation set is iterated over. But I might be wrong there, i will look into it – Djaik Navidson Mar 21 '19 at 13:40
• Could be. I don’t use generators for validation data so I’m not sure if the specifics – kbrose Mar 21 '19 at 16:19

There is nothing fundamentally wrong with your code, but maybe your model is not right for your current toy-problem.

In general, this is typical behavior when training in deep learning. Think about it, your target loss is the training loss, so it is directly affected by the training process and as you said "improving quickly". The validation loss is only affected indirectly, so naturally it will be more volatile in comparison.

When you are training, the model is attempting to estimate the real distribution of the data, however all it got is the distribution of the training dataset to rely on (which is similar but not the same).

Suggestions:

I think that your model is an over-kill for a 256x256 grayscale dataset with just 5 classes (Resnet was designed for ImageNet which contains RGB images from 1000 categories). In current state, the model finds its very easy to memorize the training set and overfit. You should look for models that are meant to be used for MNIST, or at most CIFAR10.

You can insist on this model and attempt to increase regularization, but I'm not sure if it will be enough to prevent the model from overfitting in this case.

• Thank you very much! I chose a bigger network because the data is fairly complicated (Im trying to predict electron distribution types from scattering images that vary wildly with orientation) but I will try with a smaller net as per your suggestion. Thank you again, I will report back with the results and mark the questions as solved if it works! – Djaik Navidson Mar 21 '19 at 13:36
• Thank you again! I got it to work by telling it to actually look at ALL my images (my batch size was too low). Now everything converges as I expected. The overfitting issue you brought up might still be relevant however, I will now evaluate the model on some new data and see how well it holds up compared to something simpler. Thank you again for your time! – Djaik Navidson Mar 22 '19 at 9:15