Keras ResNet-50 not performing as expected

Question

I am trying to build a neural network that is capable of classifying the make and model of a car. I am using the VMMR dataset to verify that the network is working, at which point I would like to start introducing my own data.

In the following paper A Large and Diverse Dataset for Improved Vehicle Make and Model Recognition the authors state that they used a ResNet-50 network and were able to achieve a 90.26% test accuracy using just the VMMR dataset.

I am using Tensorflow with Keras to build my network. I am using the ResNet-50 model from Keras.Applications, pre-trained on imagenet. However I was unable to reproduce results that were remotely similar (think 20-30%) to the 90% that is mentioned in the VMMR paper.

So I decided to simplify the problem and the test the network on CIFAR-10. But the network again performed worse than expected. I ran the network on the CIFAR-10 dataset both with and without data augmentation (so I could rule out that data augmentation was the problem) and achieved test accuracy's of 70% and 77%. However in the paper Deep Residual Learning for Image Recognition the authors were able to achieve a test accuracy of 93.03% with a ResNet-50 model in CIFAR-10, which is much higher.

This leads me to the following conclusion. Either there is something wrong with the Keras ResNet implementation (which seems unlikely since I cannot find anyone else with the same problem) or there is something wrong with my code.

As mentioned before, I ruled out that the problem is caused by my data augmentation. But I am also confident that the loading and preparation of the data is done correctly since I am able to classify the MNIST dataset just fine. Also when I try to classify between 10 classes from the VMMR dataset the network performs fine. Only when I try to distinguish between more classes the network does not perform well. (This is not a case of the problem being to complex for the mode to solve since ResNet-50, 101 and 152 all yield similar results)

It is especially strange that the network is able to distinguish between 10 classes from the VMMR dataset (which are resized to 224x224) but not (Meaning with only 77% test accuracy, when expecting around 93% (see above)) between 10 classes from the CIFAR-10 dataset (which are 32x32 and should be a much easier problem to solve).

One final observation is my loss. For some reason when using the Keras ResNet-50 model I get very unrealistic loss. See for example the loss from the Keras ResNet-50 model with ran for 300 epochs on the CIFAR-100 dataset.

Any insights as of why this is happening or what I am doing wrong will be greatly appreciated!

Full code below

Settings.py

import pathlib
import tensorflow as tf
import numpy as np

EPOCHS = 300
BATCH_SIZE = 16                             # Amount of images per batch
CHANNELS = 3                                # Amount of color channels
DATA_DIR = "/home/joel/datasets/vmmr"       # Directory containing the images
TEST_PERCENTAGE = 0.1                       # % of data that will be used as test data
IMG_WIDTH = 224                             # Resized with of the image
IMG_HEIGHT = 224                            # Resized height of the image
AUTOTUNE = tf.data.experimental.AUTOTUNE    # Autotune prefetch operations
DATA_DIR = pathlib.Path(DATA_DIR)           # Convert DATA_DIR to pathlib Path
CLASS_NAMES = np.array(                     # Numpy Array containing all classes
    [item.name for item in DATA_DIR.glob("*")]
)

Network.py

# %% - imports

from __future__ import absolute_import, division, print_function, unicode_literals

__import__("sys").path.append("/home/joel/projects/MMR_Net/")  # noqa: E402

import os
os.environ["TF_CPP_MIN_LOG_LEVEL"] = "2"  # noqa: E402

import tensorflow as tf
import time
import numpy as np
import datetime
import Network_Simple.utilities as utils

from tensorflow.keras.layers import (Conv2D, MaxPooling2D, Flatten, Dense, GlobalAveragePooling2D, BatchNormalization)
from Network_Simple.settings import (
    AUTOTUNE,
    DATA_DIR,
    IMG_HEIGHT,
    IMG_WIDTH,
    CHANNELS,
    CLASS_NAMES,
    EPOCHS,
    IMAGE_HEIGHT,
    IMAGE_WIDTH
)

# %% - Allow memory growth, code won't run without this!
gpu_devices = tf.config.experimental.list_physical_devices("GPU")
tf.config.experimental.set_memory_growth(gpu_devices[0], True)
tf.keras.backend.set_floatx("float32")

# %% - load data

# Load and shuffle data
files = utils.get_files(DATA_DIR)
np.random.shuffle(files)

# Split train and test data
train_array, test_array = utils.split_dataset(files)

# Convert arrays to datasets
train_ds = utils.array_to_dataset(train_array)
test_ds = utils.array_to_dataset(test_array)

# Resolve images and labels
train_ds = train_ds.map(utils.process_path, num_parallel_calls=AUTOTUNE)
test_ds = test_ds.map(utils.process_path, num_parallel_calls=AUTOTUNE)

# Resize images to desired format
train_ds = train_ds.map(utils.resize_image, num_parallel_calls=AUTOTUNE)
test_ds = test_ds.map(utils.resize_image, num_parallel_calls=AUTOTUNE)

# Augment data
train_ds = train_ds.map(utils.augment_data, num_parallel_calls=AUTOTUNE)

# Prepare datasets for training
train_ds = utils.prepare_for_training(train_ds, shuffle=True)
test_ds = utils.prepare_for_training(test_ds, shuffle=False)

# %% - Define model

# ResNet-50 implementation from Keras

model = tf.keras.Sequential(
    [
        tf.keras.applications.resnet50.ResNet50(weights="imagenet", include_top=False,
                                                input_shape=(IMG_HEIGHT, IMG_WIDTH,
                                                             CHANNELS)),
        tf.keras.layers.Flatten(),
        tf.keras.layers.Dense(len(CLASS_NAMES), activation="softmax")
    ]
)

optimizer = tf.keras.optimizers.Adam(learning_rate=0.001)

model.compile(optimizer=optimizer,
              loss='categorical_crossentropy',
              metrics=['accuracy'])

model.summary()

# %% - Create summary writers
log_dir = "logs/simple/" + datetime.datetime.now().strftime("%Y%m%d-%H%M%S")
tensorboard_callback = tf.keras.callbacks.TensorBoard(log_dir=log_dir, update_freq="batch")

# %% - Training the network
model.fit(train_ds, validation_data=test_ds, epochs=EPOCHS, callbacks=[tensorboard_callback])

utilities.py

__import__("sys").path.append("/home/joel/projects/MMR_Net/")  # noqa: E402

import tensorflow as tf
import math
import os
import tensorflow_addons as tfa
import matplotlib.pyplot as plt

from Network_Simple.settings import (
    CHANNELS,
    TEST_PERCENTAGE,
    BATCH_SIZE,
    AUTOTUNE,
    CLASS_NAMES,
    IMG_HEIGHT,
    IMG_WIDTH
    )


def get_label(file_path):
    # convert the path to a list of path components
    parts = tf.strings.split(file_path, os.path.sep)
    # The second to last is the class-directory
    return parts[-2] == CLASS_NAMES


def process_path(file_path):
    label = get_label(file_path)
    # load the raw data from the file as a string
    img = tf.io.read_file(file_path)
    # convert the compressed string to a 3D uint8 tensor
    img = tf.image.decode_jpeg(img, channels=CHANNELS)
    # Use `convert_image_dtype` to convert to floats in the [0,1] range.
    img = tf.image.convert_image_dtype(img, tf.float32)
    return img, label


def array_to_dataset(array):
    tensor = tf.convert_to_tensor(array)
    dataset = tf.data.Dataset.from_tensor_slices(tensor)
    return dataset


def split_dataset(files):

    image_count = len(files)

    # Calculate split amounts
    train_amount = int((1 - TEST_PERCENTAGE) * image_count)
    test_amount = int(image_count - train_amount)

    # Assign leftover records due to rounding
    while train_amount + test_amount < image_count:
        train_amount += 1

    # Create split arrays
    train_ds = files[:train_amount]
    test_ds = files[train_amount:]

    return train_ds, test_ds


def prepare_for_training(ds, shuffle=True, shuffle_buffer_size=100):

    if shuffle:
        ds = ds.shuffle(buffer_size=shuffle_buffer_size)

    ds = ds.batch(BATCH_SIZE)
    ds = ds.prefetch(buffer_size=AUTOTUNE)

    return ds


def get_files(start_dir):

    dirs = []
    files = []

    for dir in start_dir.glob("*"):
        dir_files = []
        for filename in dir.iterdir():
            dir_files.append(str(filename))

        dirs.append(dir)
        files += dir_files

    return files


def resize_image(image, label):
    # resize the image to the desired size.
    image = tf.image.resize_with_pad(image, IMG_HEIGHT, IMG_WIDTH)
    # Use `convert_image_dtype` to convert to floats in the [0,1] range.
    image = tf.image.convert_image_dtype(image, tf.float32)
    return image, label


def augment_data(image, label):

    # Horizontal flip (always 50% chance)
    if(tf.random.uniform([]) < 0.5):
        image = tf.image.flip_left_right(image)

    # ================================
    # === Geometric augmentations ====
    # ================================

    geo_rand = tf.random.uniform([], minval=0, maxval=3, dtype=tf.dtypes.int32)

    # Random crop
    if(geo_rand == 1):
        # TODO: Scale crop accordingly!!
        image = tf.image.random_crop(image, size=[200, 200, 3])
        image = tf.image.resize(image, size=[IMG_HEIGHT, IMG_WIDTH])

    # Rotate left/right (10 to 20 degrees)
    if(geo_rand == 2):
        degrees = tf.random.uniform([], minval=10, maxval=20, dtype=tf.dtypes.int32)

        if(tf.random.uniform([]) < 0.5):
            degrees *= -1

        image = tfa.image.rotate(image, (math.pi / 180.0 * float(degrees)),
                                 interpolation="BILINEAR")

    # JPEG Noise
    if(geo_rand == 3):
        image = tf.image.random_jpeg_quality(image,
                                             min_jpeg_quality=25,
                                             max_jpeg_quality=35)

    # TODO: Random earasing
    if(geo_rand == 4):
        pass

    # =================================
    # === Photometric augmentations ===
    # =================================

    photo_rand = tf.random.uniform([], minval=0, maxval=4, dtype=tf.dtypes.int32)

    # Constrast
    if(photo_rand == 1):
        image = tf.image.random_contrast(image, 0.3, 2)

    # Saturation
    if(photo_rand == 2):
        image = tf.image.random_saturation(image, 2, 4)

    # Hue
    if(photo_rand == 3):
        image = tf.image.random_hue(image, 0.5)

    # Brightness
    if(photo_rand == 4):
        image = tf.image.random_brightness(image, 0.4)

    return image, label

EDIT - Incorporated changes suggested by Matias Valdenegro

• Switched optimizer to SGD with learning rate=0.1, momentum=0.9 and decay=0.0001. I also implemented the learning rate scheduler. I converted the interations to epochs and set the learning rate to 0.01, reducing it by a factor of ten at epoch 80 and again at epoch 120.

optimizer = tf.keras.optimizers.SGD(learning_rate=0.1, momentum=0.9, decay=0.0001)

def scheduler(epoch):
    epoch = int(epoch)

    if epoch < 80:
        learning_rate = 0.1
    elif epoch < 120:
        learning_rate = 0.01
    else:
        learning_rate = 0.001

    print(f"Set base learning rate for epoch {epoch + 1}: {learning_rate}")

    return learning_rate

• Changed data augmentation to match the paper

def augment_data(image, label):

    # Pad 4 pixels on each size
    padded_height = IMG_HEIGHT + 8
    padded_width = IMG_WIDTH + 8

    image = tf.image.resize_with_crop_or_pad(image, padded_height, padded_width)

    # Randomly crop the padded image back to the original size
    image = tf.image.random_crop(image, size=[IMG_HEIGHT, IMG_WIDTH, CHANNELS])

    return image, label

• Changed batch size to 128

• Changed epochs to 165.

However I still do not get similar results. The loss does seems to be a lot more realistic though. Is there anything else I might be overlooking?

See below for the results of the above mentioned configuration on the CIFAR-10 dataset.

(No further improvements after 100 epochs )

Answer 1

When implementing a model from a paper to reproduce their results, it is very important to pay attention to all the details. For this case there are some important differences when comparing to the CIFAR10 results of ResNet:

• You are using the Adam optimizer, while the ResNet paper uses SGD with a learning rate schedule. Adam is known to have issues converging to the best solution, and generally a well tuned SGD outperforms it. For reference this is the learning rate scheduler used by the paper:

We start with a learning rate of 0.1, divide it by 10 at 32k and 48k iterations, and terminate training at 64k iterations

This schedule can be implemented with the keras callback LearningRateScheduler

• You are not using the same data augmentation as in the ResNet paper. Quoting from the paper:

We follow the simple data augmentation in [24] for training: 4 pixels are padded on each side, and a 32×32 crop is randomly sampled from the padded image or its horizontal flip. For testing, we only evaluate the single view of the original 32×32 image.

• You are using a batch size of 16, while the paper uses a batch size of 128. This is critical as batch size controls how noisy is the gradient, and can radically change the solution learned by SGD.

• Epochs looks okay, the paper trains for approximately 165 epochs (they use iterations which depends on batch size), so you might be overtraining the model.

After fixing all of these issues you might get closer to the accuracy/error reported by the ResNet paper. For reference the Keras examples do contain a ResNet sample using CIFAR10 that gets close results here.