2
$\begingroup$

New to machine learning and tried to train my bird recognization model and found very high validation loss and inaccuracy.

I'm using this dataset: http://www.vision.caltech.edu/visipedia/CUB-200-2011.html

Is my model over-fitting? What can I do to fix it?

here is the graphs enter image description here

enter image description here

and here is my code.

def train_CNN(train_directory, target_size=(200, 200), classes=None,
              batch_size=128, num_epochs=20, num_classes=5, verbose=0, show_graph=False):

    CHECKPOINT_DIRECTORY = './checkpoints'
    SAVE_CHECKPOINT_PATH = CHECKPOINT_DIRECTORY + \
        '/{epoch:02d}_{val_acc:.4f}.h5'
    if not os.path.exists(CHECKPOINT_DIRECTORY):
            os.makedirs(CHECKPOINT_DIRECTORY)

    train_datagen = ImageDataGenerator(rescale=1. / 255, validation_split=0.1)

    train_generator = train_datagen.flow_from_directory(
        train_directory,  # This is the source directory for training images
        target_size=target_size,  # All images will be resized to 200 x 200
        batch_size=batch_size,
        classes=classes,
        subset='training',
        class_mode='categorical')

    val_generator = train_datagen.flow_from_directory(
        train_directory,  
        target_size=target_size,  # All images will be resized to 200 x 200
        batch_size=batch_size,
        classes=classes,
        subset='validation',
        class_mode='categorical')

    input_shape = tuple(list(target_size)+[3])

    # Model architecture
    model = tf.keras.models.Sequential([
        # Note the input shape is the desired size of the image 200x 200 with 3 bytes color
        # The first convolution
        tf.keras.layers.Conv2D(
            16, (3, 3), activation='relu', input_shape=input_shape),
        tf.keras.layers.MaxPooling2D(2, 2),
        # The second convolution
        tf.keras.layers.Conv2D(32, (3, 3), activation='relu'),
        tf.keras.layers.MaxPooling2D(2, 2),
        # The third convolution
        tf.keras.layers.Conv2D(64, (3, 3), activation='relu'),
        tf.keras.layers.MaxPooling2D(2, 2),
        # The fourth convolution
        tf.keras.layers.Conv2D(64, (3, 3), activation='relu'),
        tf.keras.layers.MaxPooling2D(2, 2),
        # The fifth convolution
        tf.keras.layers.Conv2D(64, (3, 3), activation='relu'),
        tf.keras.layers.MaxPooling2D(2, 2),
        # Flatten the results to feed into a dense layer
        tf.keras.layers.Flatten(),
        # 512 neuron in the fully-connected layer
        tf.keras.layers.Dense(512, activation='relu'),

        tf.keras.layers.Dense(num_classes, activation='softmax')
    ])

    # Optimizer and compilation
    model.compile(loss='categorical_crossentropy',
                  optimizer=RMSprop(lr=0.001),
                  metrics=['accuracy'])

    # Create a callback that saves the model's weights
    model_checkpoint_callback = tf.keras.callbacks.ModelCheckpoint(filepath=SAVE_CHECKPOINT_PATH,
                                                     save_weights_only=False,
                                                     save_best_only=True,
                                                     monitor='val_acc', 
                                                     mode='max',  # related to the value of monitor
                                                     verbose=1)

    tensorboard_callback = tf.keras.callbacks.TensorBoard(
        log_dir='./logs/',
        histogram_freq=1,
        batch_size=batch_size)

    reduce_lr_callback = tf.keras.callbacks.ReduceLROnPlateau(
        monitor='val_loss',
        factor=0.5,
        patience=3,
        min_lr=1e-6)


    INITIAL_EPOCH = 0

    # Training
    history = model.fit_generator(
            train_generator,
            steps_per_epoch=train_generator.samples // batch_size, #int(total_sample/batch_size),
            validation_data=val_generator,
            validation_steps=val_generator.samples // batch_size,
            epochs=num_epochs,
            verbose=verbose,
            initial_epoch= INITIAL_EPOCH,
            callbacks=[model_checkpoint_callback, tensorboard_callback, reduce_lr_callback])

    if show_graph == True:
        visualizeTraining(history)

    return model


def visualizeTraining(history):
  graphFolder = 'graph'
  graphViz = graphFolder + '/graph.jpeg'
  graphVizLoss = graphFolder + '/loss.jpeg'
  if not os.path.exists(graphFolder):
    os.makedirs(graphFolder)
  plt.figure()
  plt.plot(history.history['acc'])
  plt.plot(history.history['val_acc'])
  plt.title('Model accuracy')
  plt.ylabel('Accuracy')
  plt.xlabel('Epoch')
  plt.legend(['Train', 'Validation'], loc='upper left')
  plt.savefig(graphViz)

  # Plot training & validation loss values
  plt.figure()
  plt.plot(history.history['loss'])
  plt.plot(history.history['val_loss'])
  plt.title('Model loss')
  plt.ylabel('Loss')
  plt.xlabel('Epoch')
  plt.legend(['Train', 'Validation'], loc='upper left')
  plt.savefig(graphVizLoss)

here is how I call it

target_size = (200, 200)
CLASSES = getClassLines(CLASSES_FILE)
model = train_CNN(IMAGES_DIR, target_size, CLASSES, 128, 30, 200, 1, True)
$\endgroup$

3 Answers 3

1
$\begingroup$

Training loss goes to zero while validation loss increasing is a clear sign of overfitting - similarly, accuracy results also indicate overfitting.

I would try simplifying the model a little bit: Just 2 layers of Conv-MaxPool pairs would be a good starting point, each with 128 filters perhaps? And maybe a dense layer with 64 or 128 neurons after those. You seem to have too many parameters in your model.

And finally, if your aim is to do image classification, you might want to look into "transfer learning".

$\endgroup$
4
  • $\begingroup$ thanks @serall, which dense layer should go 1st? the 64 or 128 and why? $\endgroup$
    – Franva
    Oct 27, 2019 at 13:44
  • $\begingroup$ also, where to learn what kind of layers should be used in which conditions and why? $\endgroup$
    – Franva
    Oct 27, 2019 at 13:45
  • $\begingroup$ Reducing the model will reduce the overfitting but it is also likely decrease accuracy. The best approach is to increase regularization, a good choice would be applying dropout to the convolutional layers. I am not a keras user but in pytorch there are two kinds of dropout, be sure you use a dropout that drops entire feature maps or it won't work properly for conv layers. $\endgroup$ Oct 27, 2019 at 14:02
  • $\begingroup$ @Franva: I would try a single 128 dense layer at first and change the number of neurons or the number of filters in the convolutional layers by trial and error. $\endgroup$
    – serali
    Oct 27, 2019 at 14:15
1
$\begingroup$

You could also maybe try to implement dropout, as part of additional strategies to prevent overfitting

$\endgroup$
0
$\begingroup$

The answer probably has something to do with the fact that your train and test accuracy start at 0.0, which is abnormal. You have 5 classes, so accuracy should start at 0.2. Your model doesn't appear to be the problem, you made a mistake somewhere.

$\endgroup$
1
  • $\begingroup$ Thanks Nicolas, but not 5 classes, I have 200 classes. the 5 is just the default value for that parameter. I think you missed the piece of code below that big piece of code. :) $\endgroup$
    – Franva
    Oct 29, 2019 at 6:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.