3
$\begingroup$

I followed this basic classification TensorFlow tutorial using the Fashion MNIST dataset. The training set contains 60,000 28x28 pixels greyscale images, split into 10 classes (trouser, pullover, shoe, etc...). The tutorial uses a simple model:

model = keras.Sequential([
    keras.layers.Flatten(input_shape=(28, 28)),
    keras.layers.Dense(128, activation='relu'),
    keras.layers.Dense(10)
])

This model reaches 91% accuracy after 10 epochs.

I am now practicing with another dataset called CIFAR-10, which consists of 50,000 32*32 pixels RGB images, also split into 10 classes (frog, horse, boat, etc...).

Considering that both the Fashion MNIST and CIFAR-10 datasets are pretty similar in terms of number of images and image size and that they have the same number of classes, I naively tried training a similar model, simply adjusting the input shape:

  model = keras.Sequential([
     keras.layers.Flatten(input_shape=(32, 32, 3)),
     keras.layers.Dense(128, activation='relu'),
     keras.layers.Dense(10)
  ])

Alas, after 10 epochs, the model reaches an accuracy of 45%. What am I doing wrong?

I am aware that I have thrice as many samples in an RGB image than in a grayscale image, so I tried increasing the number of epochs as well as the size of the intermediate dense layer, but to no avail.


Below is my full code:

import tensorflow as tf
import IPython.display as display
from PIL import Image
from tensorflow import keras
import numpy as np
import matplotlib.pyplot as plt
import pdb
import pathlib
import os
from tensorflow.keras import layers #Needed to make the model
from tensorflow.keras import datasets, layers, models

(train_images, train_labels), (test_images, test_labels) = datasets.cifar10.load_data()

IMG_HEIGHT = 32
IMG_WIDTH = 32

class_names = ['airplane', 'automobile', 'bird', 'cat', 'deer',
               'dog', 'frog', 'horse', 'ship', 'truck']


train_images = train_images / 255.0
test_images = test_images / 255.0

def make_model():
      model = keras.Sequential([
         keras.layers.Flatten(input_shape=(IMG_HEIGHT, IMG_WIDTH, 3)),
         keras.layers.Dense(512, activation='relu'),
         keras.layers.Dense(10)
      ])
      model.compile(optimizer='adam',
                   loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
                   metrics=['accuracy'])
      return model

model=make_model()
history = model.fit(train_images, train_labels, epochs=10)
$\endgroup$
  • $\begingroup$ The accuracy that you report, is it the training or testing accuracy? In general, have you checked whether your model can reach a very high training accuracy on CIFAR-10 after training it long enough? If it is not the case, then your model is too simple and you should add more parameters. I have no source for this but from experience I would say CIFAR-10 is much more complex to learn than MNIST. $\endgroup$ – mrzo Apr 5 at 9:30
  • $\begingroup$ I am reporting the training accuracy, since I am looking at the output of history = model.fit(train_images, train_labels, epochs=10). $\endgroup$ – Sheldon Apr 5 at 10:19
  • $\begingroup$ You are not doing anything wrong, model that works with one dataset does not mean it will work in another dataset. MNIST datasets are much easier to classify than CIFAR10. $\endgroup$ – Dr. Snoopy Apr 5 at 11:23
  • $\begingroup$ Sorry about that! I am OK to delete either post, but I received interesting answers on both SO and the Data Science Stack Exchange and I would like to keep track of them all. What should I do? $\endgroup$ – Sheldon Apr 5 at 19:02
2
$\begingroup$

Your model is not sufficiently complex to adequately classify the CIFAR 10 data set. CIFAR-10 is considerably more complex than the Fashion-MNIST data set and therefore you need a more complex model.You can add more hidden layers to your model to achieve this. You should also add DROPOUT layers to prevent over fitting. Perhaps the easiest solution is to use transfer learning. I would recommend using the MobileNet CNN model if you want to try transfer learning. Documentation for that can be found here. Since CIFAR-10 has 50,000 sample images I do not think you will need data augmentation. First try a more complex model without augmentation and see what accuracy you achieve. If it is not adequate then use the keras ImageData Generator to provide data augmentation. Documentation for that is here.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks for your reply Gerry. I will first give CNN a shot, then I will go for transfer learning! $\endgroup$ – Sheldon Apr 5 at 17:14
2
$\begingroup$

I'm using this model (basically building on work of Chollet). It uses a pretrained model (VGG16) for a multiclass image recognition problem.

from keras.applications import VGG16
import os, datetime
import numpy as np
from keras.preprocessing.image import ImageDataGenerator
from keras.utils import to_categorical
from keras import models, layers, optimizers, regularizers
from keras.callbacks import EarlyStopping
from keras.callbacks import ReduceLROnPlateau
from keras.layers.core import Dense, Dropout, Activation
from keras.layers.normalization import BatchNormalization
from PIL import ImageFile
import statistics
ImageFile.LOAD_TRUNCATED_IMAGES = True

###############################################
# DIR with training images
base_dir = 'C:/pathtoimages'
# Number training images
ntrain = 2000
# Number validation images
nval  = 500
# Batch size
batch_size = 20 #20
# Epochs (fine tuning [100])
ep = 400 #400
# Epochs (first step [30])
ep_first = 30 
# Number of classes (for training, output layer)
nclasses = 30
###############################################
start = datetime.datetime.now()

conv_base = VGG16(weights='imagenet', include_top=False, input_shape=(150, 150, 3))
train_dir = os.path.join(base_dir, 'train')
validation_dir = os.path.join(base_dir, 'val')
#test_dir = os.path.join(base_dir, 'test')

datagen = ImageDataGenerator(rescale=1./255)

def extract_features(directory, sample_count):
    features = np.zeros(shape=(sample_count, 4, 4, 512))
    labels = np.zeros(shape=(sample_count))
    generator = datagen.flow_from_directory(
        directory,
        target_size=(150, 150),
        batch_size=batch_size,
        class_mode='binary')
    i = 0
    for inputs_batch, labels_batch in generator:
        features_batch = conv_base.predict(inputs_batch)
        features[i * batch_size : (i + 1) * batch_size] = features_batch
        labels[i * batch_size : (i + 1) * batch_size] = labels_batch
        i += 1
        if i * batch_size >= sample_count:
            break
    return features, labels

train_features, train_labels = extract_features(train_dir, ntrain)
validation_features, validation_labels = extract_features(validation_dir, nval)
#test_features, test_labels = extract_features(test_dir, 1000)

# Labels and features
train_labels = to_categorical(train_labels)
validation_labels = to_categorical(validation_labels)
#test_labels = to_categorical(test_labels)
train_features = np.reshape(train_features, (ntrain, 4 * 4 * 512))
validation_features = np.reshape(validation_features, (nval, 4 * 4 * 512))
#test_features = np.reshape(test_features, (1000, 4 * 4 * 512))

#######################################
# Model
model = models.Sequential()
model.add(conv_base)
model.add(layers.Flatten())
model.add(layers.Dense(4096, activation='relu',kernel_regularizer=regularizers.l2(0.003)))#0.002
model.add(BatchNormalization())

model.add(layers.Dense(2048, activation='relu',kernel_regularizer=regularizers.l2(0.003)))#0.002
model.add(layers.Dense(2048, activation='relu',kernel_regularizer=regularizers.l2(0.003)))#0.002
model.add(BatchNormalization())

model.add(layers.Dense(1024, activation='relu',kernel_regularizer=regularizers.l2(0.003)))#0.002
model.add(layers.Dense(1024, activation='relu',kernel_regularizer=regularizers.l2(0.003)))#0.002
model.add(BatchNormalization())

model.add(layers.Dense(512, activation='relu',kernel_regularizer=regularizers.l2(0.003)))#0.002
model.add(layers.Dense(512, activation='relu',kernel_regularizer=regularizers.l2(0.003)))#0.002
model.add(BatchNormalization())

model.add(layers.Dense(256, activation='relu',kernel_regularizer=regularizers.l2(0.003)))#0.002
model.add(layers.Dense(256, activation='relu',kernel_regularizer=regularizers.l2(0.003)))#0.002
model.add(BatchNormalization())

model.add(layers.Dense(128, activation='relu',kernel_regularizer=regularizers.l2(0.003)))#0.002
model.add(layers.Dense(128, activation='relu',kernel_regularizer=regularizers.l2(0.003)))#0.002
model.add(layers.Dense(128, activation='relu',kernel_regularizer=regularizers.l2(0.003)))#0.002
model.add(layers.Dense(128, activation='relu',kernel_regularizer=regularizers.l2(0.003)))#0.002

model.add(layers.Dense(nclasses, activation='softmax'))
conv_base.trainable = False

#######################################
# Data generators
train_datagen = ImageDataGenerator(
      rescale=1./255,
      rotation_range=40,
      width_shift_range=0.2,
      height_shift_range=0.2,
      shear_range=0.2,
      zoom_range=0.2,
      horizontal_flip=True,
      fill_mode='nearest')

# Note that the validation data should not be augmented!
test_datagen = ImageDataGenerator(rescale=1./255)

train_generator = train_datagen.flow_from_directory(
        # This is the target directory
        train_dir,
        # All images will be resized to 150x150
        target_size=(150, 150),
        batch_size=batch_size,
        # Since we use categorical_crossentropy loss, we need binary labels
        class_mode='categorical')

validation_generator = test_datagen.flow_from_directory(
        validation_dir,
        target_size=(150, 150),
        batch_size=batch_size,
        class_mode='categorical')

# Model compile / fit
model.compile(loss='categorical_crossentropy',
              optimizer=optimizers.RMSprop(lr=2e-5),
              metrics=['acc'])

# early stopping: https://keras.io/callbacks/#earlystopping
es = EarlyStopping(monitor='val_loss', mode='min', min_delta=0.001, verbose=1, patience=40, restore_best_weights=True)
reduce_lr = ReduceLROnPlateau(monitor='val_loss', mode='min', factor=0.9, patience=15, min_lr=1e-20, verbose=1, cooldown=3)

history = model.fit_generator(
      train_generator,
      steps_per_epoch=round((ntrain+nval)/batch_size,0),
      epochs=ep_first,
      validation_data=validation_generator,
      validation_steps=20, #50
      verbose=2,
      callbacks=[es, reduce_lr])

#######################################
# Fine tuning
conv_base.trainable = True

set_trainable = False
for layer in conv_base.layers:
    if layer.name == 'block5_conv1':
        set_trainable = True
    if set_trainable:
        layer.trainable = True
    else:
        layer.trainable = False

model.compile(loss='categorical_crossentropy',
              optimizer=optimizers.RMSprop(lr=0.00001), #1e-5
              metrics=['acc'])

history = model.fit_generator(
      train_generator,
      steps_per_epoch=round((ntrain+nval)/batch_size,0),
      epochs=ep,
      validation_data=validation_generator,
      validation_steps=20,
      callbacks=[es, reduce_lr])

#######################################
# Save model
model.save('C:/yourpath/yourmodel.hdf5')
end = datetime.datetime.now()
delta = str(end-start)

# Metrics
acc = history.history['acc']
acc = acc[-5:]
val_acc = history.history['val_acc']
val_acc = val_acc[-5:]
loss = history.history['loss']
loss = loss[-5:]
val_loss = history.history['val_loss']
val_loss = val_loss[-5:]

# End statement
print("============================================")
print("Time taken (h/m/s): %s" %delta[:7])
print("============================================")
print("Metrics (average last five steps)")
print("--------------------------------------------")
print("Loss       %.3f" %statistics.mean(loss))
print("Val. Loss  %.3f" %statistics.mean(val_loss))
print("--------------------------------------------")
print("Acc.       %.3f" %statistics.mean(acc))
print("Val. Acc.  %.3f" %statistics.mean(val_acc))
print("============================================")
print("Epochs:    %s / %s" %(ep,ep_first))
| improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks, Peter! I will give it a shot! I wanted to build a model from scratch, but maybe pre-trained models will give better results. $\endgroup$ – Sheldon Apr 5 at 17:09
  • $\begingroup$ Using pretrained weights will likely perform better. However, you can also just skip the pretrained part and build your own model around the data generator. Have a look here: github.com/fchollet/deep-learning-with-python-notebooks/blob/… $\endgroup$ – Peter Apr 5 at 17:15
1
$\begingroup$

Two things come to mind:

You can add a data generator. This will generate new images from your current images by introducing a bunch of small changes (i.e. randomly rotating, zooming, shearing, shifting horizontally/vertically...), forcing the model to learn important distinguishing features between the different classes of images.

You can also add dropout layers to combat overfitting.

Here is a good example: https://keras.io/examples/cifar10_cnn/

| improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ Thanks for your reply, Derek! I agree that either performing data augmentation or using CNN will help, but ultimately I would like to know why I cannot simply use the model that worked well on the Fashion MNIST dataset. $\endgroup$ – Sheldon Apr 5 at 8:51
  • $\begingroup$ I think because CIFAR-10 is simply a harder classification problem than Fashion-MNIST (or regular MNIST for that matter). The types of images within each class in CIFAR-10 vary more widely (i.e. there are lots of different types of dogs viewed from a variety of angles, and it's harder for the model to learn what makes a dog a dog than what makes a shirt a shirt). This means that the same model you used for Fashion-MNIST won't achieve the same accuracy for CIFAR-10 $\endgroup$ – Derek O Apr 5 at 9:48
1
$\begingroup$

I think your model is not complex enough to learn from the CIFAR-10 datasets.

You can find CIFAR-10 classification datasets results using different models and activation functions here.

Looking from the results, I can see that you will need to use a dense CNN model with Exponential Linear units (ELU) to get better accuracy.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks for your reply. I will go for CNN, then! By the way, thanks for sharing the link to Rodrigob's github page: it is great that someone bothered compiling the performances of different algorithms applied to the same dataset. $\endgroup$ – Sheldon Apr 5 at 13:38
1
$\begingroup$

Since you just achieve a training accuracy of 45%, I assume that your model is too simple. What you can do:

1) Use more hidden layers: more hidden layers increase the number of parameters and complexity of your model. However, since you are using dense, fully-connected layers you might see that your model gets big and slow pretty quickly. Therefore, I would suggest:

2) Use Convolutional layers. They are made for image classification since they allow much more efficient usage of parameters and training of more hidden layers.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks for your reply. I will avoid using dense layers for my future image recognition projects, and will use CNN instead! $\endgroup$ – Sheldon Apr 5 at 13:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.