I'm fairly new at working with neural networks and think I am making some basic mistake. I am trying to assign simulated images to 5 classes to test, what (if any) networks are helpful for a large data problem we have in our group. I am training the ResNet50 CNN included in Keras on 40000 (256 by 256 pixel greyscale) images. While training loss is improving quickly within the first epoch, validation loss fluctuates wildly in (to me) fairly random manner. enter image description here

enter image description here

I am trying to use as many high level functions as possible and have ended up with the following code:

from keras.preprocessing.image import ImageDataGenerator
from keras.applications.resnet50 import ResNet50
from keras.models import Model
from keras.layers import GlobalAveragePooling2D, Dense

inputShape = (256, 256,1)
targetSize = (256, 256  )
batchSize = 128

# First, load resnet
base_model = ResNet50(include_top =False,
                weights =None,
x = base_model.output

# modified based on the article https://github.com/priya-dwivedi/
# Deep-Learning/blob/master/resnet_keras/Residual_Networks_yourself.ipynb

x = GlobalAveragePooling2D()(x)
predictions = Dense(5, activation= 'softmax')(x)
model = Model(inputs = base_model.input, outputs = predictions)

# Compile model, might want to change loss and metrics
        metrics = ["acc"]

# Define the generators to read the data. Training generator performs data argumentation as well
train_datagen = ImageDataGenerator(
        samplewise_center =True,
        samplewise_std_normalization = True,

test_datagen = ImageDataGenerator(
        samplewise_center =True,
        samplewise_std_normalization = True,

train_generator = train_datagen.flow_from_directory(
        color_mode = 'grayscale',
        save_to_dir = 'trainingData/argumentedImages',

validation_generator = test_datagen.flow_from_directory(
        color_mode = 'grayscale',

# Fit the Modell, saving the history for later plotting
history = model.fit_generator(
        #steps_per_epoch = 62,
        steps_per_epoch = 31,

I can always create more images or train longer, but to me, this looks as if something fundamental went wrong somewhere. I would be very gratefull for any and all ideas. Thank you!

EDIT: I was told to stress that validation and training set are both created by exaclty the same simulation routine, so they should be relativly easy to classify

EDIT2: Found the error! My batch size did not fit with the amount of data and the steps by epoch, resulting in the CNN not seeing all the training data. Now everything converges nicely and I can evaluate the choice of modell. Thanks to all contributers

  • $\begingroup$ Are you computing validation metrics on only batchSize examples? Maybe you should use more examples to get a more robust estimate of those metrics? $\endgroup$
    – kbrose
    Mar 21, 2019 at 13:20
  • $\begingroup$ As far I understand, "validation_steps=9" in model.fit_generator should make it so that the entire validation set is iterated over. But I might be wrong there, i will look into it $\endgroup$ Mar 21, 2019 at 13:40
  • $\begingroup$ Could be. I don’t use generators for validation data so I’m not sure if the specifics $\endgroup$
    – kbrose
    Mar 21, 2019 at 16:19

1 Answer 1


There is nothing fundamentally wrong with your code, but maybe your model is not right for your current toy-problem.

In general, this is typical behavior when training in deep learning. Think about it, your target loss is the training loss, so it is directly affected by the training process and as you said "improving quickly". The validation loss is only affected indirectly, so naturally it will be more volatile in comparison.

When you are training, the model is attempting to estimate the real distribution of the data, however all it got is the distribution of the training dataset to rely on (which is similar but not the same).


I think that your model is an over-kill for a 256x256 grayscale dataset with just 5 classes (Resnet was designed for ImageNet which contains RGB images from 1000 categories). In current state, the model finds its very easy to memorize the training set and overfit. You should look for models that are meant to be used for MNIST, or at most CIFAR10.

You can insist on this model and attempt to increase regularization, but I'm not sure if it will be enough to prevent the model from overfitting in this case.

  • 1
    $\begingroup$ Thank you very much! I chose a bigger network because the data is fairly complicated (Im trying to predict electron distribution types from scattering images that vary wildly with orientation) but I will try with a smaller net as per your suggestion. Thank you again, I will report back with the results and mark the questions as solved if it works! $\endgroup$ Mar 21, 2019 at 13:36
  • 1
    $\begingroup$ Thank you again! I got it to work by telling it to actually look at ALL my images (my batch size was too low). Now everything converges as I expected. The overfitting issue you brought up might still be relevant however, I will now evaluate the model on some new data and see how well it holds up compared to something simpler. Thank you again for your time! $\endgroup$ Mar 22, 2019 at 9:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.