2
$\begingroup$

I'm new to Tensorflow and Keras and I some background knowledge of how CNN's work. I'm using a basic sequential model based on the code by https://pythonprogramming.net/convolutional-neural-network-deep-learning-python-tensorflow-keras/

I have a problem where my results variation is very big. The first time I ran the model today I got around 90% accuracy. But the runs after that were around 25% which is as good as guessing since I have four classes.

The results in Tensorboard

Here's my code:

tf.reset_default_graph()

batch_size = 32

logdir = os.path.join("logs", datetime.datetime.now().strftime("%Y%m%d-%H%M%S"))
tensorboard_callback = tf.keras.callbacks.TensorBoard(logdir, histogram_freq=1)

X = X/255.0

model = Sequential()

model.add(Conv2D(64, (3, 3), input_shape=X.shape[1:]))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))

model.add(Conv2D(64, (3, 3)))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))

model.add(Flatten())  # this converts our 3D feature maps to 1D feature vectors
model.add(Dense(64))
model.add(Activation('relu'))

model.add(Dense(4))
model.add(Activation('softmax'))

model.compile(loss='categorical_crossentropy',
          optimizer='adam',
          metrics=['accuracy'])


with tf.Session() as sess:
model.fit(train_X, train_y, batch_size=batch_size, epochs=3, 
validation_split=0.1, callbacks=[tensorboard_callback], validation_data=(test_X, test_y))

Am I doing something completely wrong with the model? I do have quite a small dataset, just 1640 images. But why do some runs perform so good then?

$\endgroup$
6
  • $\begingroup$ your validation split is low (10%), this could be an issue. Did you fix the random state/validation set? It can be that you simply had a lucky shot with your val data... $\endgroup$
    – Peter
    May 28, 2019 at 13:41
  • $\begingroup$ since you have a very small sample, you could also consider training on top of a pretrained model. $\endgroup$
    – Peter
    May 28, 2019 at 13:52
  • $\begingroup$ I think I do sometimes get lucky shots.. as I'm speaking I get better results. Would it be better to increase the size of the validation set? What do you mean by fixing the state/validation set? I will look in to pretrained models (never really heard of it). $\endgroup$ May 28, 2019 at 14:36
  • $\begingroup$ Thanks for the replies btw! :) $\endgroup$ May 28, 2019 at 14:37
  • $\begingroup$ At the moment I'm reaching 100% accuracy, that doesn't seem good as well.. $\endgroup$ May 28, 2019 at 14:40

1 Answer 1

1
$\begingroup$

I commented previously, now some idea as an answer:

1) I think that the validation data may include "lucky shots" which match the trained patterns well in some cases. Also a small validation set can be a problem. As far as I know, validation data is the last X% of the data (in Keras). You can set shuffle=True to mix validation data or False to not mix it (as I understand). I would try this. https://github.com/keras-team/keras/issues/597

2) An option to get a good idea of how the model performs is cross validation. If you can afford it, train 3 or 5 or 10 models with different validation data and have a look at the average accuracy. Should give you also an idea of how volatile the results are.

3) You have a small sample. Thus, adding more validation data can be a problem. However, I would also try with 20%.

4) Since you have a small sample, the NN might have trouble getting all the relevant features. Instead of training a convnet from scratch, you can use a pretrained model. This has already learned a lot of features. Given a small sample, this could be THE way to go. Note that there are a number of different pretrained models which you can try. https://keras.io/applications/

5) Get more data (if possible).

I recently used a pretrained model in a multiclass setting. Here is my code, which is a slightly modified version of a tutorial code by F. Chollet (https://github.com/fchollet/deep-learning-with-python-notebooks/blob/master/5.3-using-a-pretrained-convnet.ipynb).

from keras.applications import VGG16
import os, datetime, statistics
import numpy as np
from keras.preprocessing.image import ImageDataGenerator
from keras.utils import to_categorical
from keras import models
from keras import layers
from keras import optimizers
from keras.layers.core import Dense, Dropout, Activation
from PIL import ImageFile
ImageFile.LOAD_TRUNCATED_IMAGES = True

###############################################
# DIR with training images
base_dir = 'C:/kerasimages'
# Number training images
ntrain = 2000
# Number validation images
nval  = 500
# Batch size
batch_size = 20
# Epochs fine tuning
ep = 600
# Epochs first step
ep_first = 50
# Number of classes (for training, output layer)
nclasses = 22
###############################################

conv_base = VGG16(weights='imagenet', include_top=False, input_shape=(150, 150, 3))
train_dir = os.path.join(base_dir, 'train')
validation_dir = os.path.join(base_dir, 'val')

datagen = ImageDataGenerator(rescale=1./255)

def extract_features(directory, sample_count):
    features = np.zeros(shape=(sample_count, 4, 4, 512))
    labels = np.zeros(shape=(sample_count))
    generator = datagen.flow_from_directory(
        directory,
        target_size=(150, 150),
        batch_size=batch_size,
        class_mode='binary')
    i = 0
    for inputs_batch, labels_batch in generator:
        features_batch = conv_base.predict(inputs_batch)
        features[i * batch_size : (i + 1) * batch_size] = features_batch
        labels[i * batch_size : (i + 1) * batch_size] = labels_batch
        i += 1
        if i * batch_size >= sample_count:
            break
    return features, labels

# Lables and features
train_features, train_labels = extract_features(train_dir, ntrain)
validation_features, validation_labels = extract_features(validation_dir, nval)
train_labels = to_categorical(train_labels)
validation_labels = to_categorical(validation_labels)
train_features = np.reshape(train_features, (ntrain, 4 * 4 * 512))
validation_features = np.reshape(validation_features, (nval, 4 * 4 * 512))

#######################################
# Model
model = models.Sequential()
model.add(conv_base)
model.add(layers.Flatten())
model.add(layers.Dense(2048, activation='relu'))#256
model.add(Dropout(0.20))
model.add(layers.Dense(1024, activation='relu'))#256
model.add(Dropout(0.20))
model.add(layers.Dense(512, activation='relu'))#256
model.add(Dropout(0.20))
model.add(layers.Dense(nclasses, activation='softmax'))
conv_base.trainable = False

#######################################
# Data generators
train_datagen = ImageDataGenerator(
      rescale=1./255,
      rotation_range=40,
      width_shift_range=0.2,
      height_shift_range=0.2,
      shear_range=0.2,
      zoom_range=0.2,
      horizontal_flip=True,
      fill_mode='nearest')

test_datagen = ImageDataGenerator(rescale=1./255)

train_generator = train_datagen.flow_from_directory(
        train_dir,
        target_size=(150, 150),
        batch_size=batch_size,
        class_mode='categorical')

validation_generator = test_datagen.flow_from_directory(
        validation_dir,
        target_size=(150, 150),
        batch_size=batch_size,
        class_mode='categorical')

# Model compile / fit
model.compile(loss='categorical_crossentropy',
              optimizer=optimizers.RMSprop(lr=2e-5),
              metrics=['acc'])

history = model.fit_generator(
      train_generator,
      steps_per_epoch=100,
      epochs=ep_first,
      validation_data=validation_generator,
      validation_steps=50,
      verbose=2)

#######################################
# Fine tuning
conv_base.trainable = True

set_trainable = False
for layer in conv_base.layers:
    if layer.name == 'block5_conv1':
        set_trainable = True
    if set_trainable:
        layer.trainable = True
    else:
        layer.trainable = False

model.compile(loss='categorical_crossentropy',
              optimizer=optimizers.RMSprop(lr=1e-5),
              metrics=['acc'])

history = model.fit_generator(
      train_generator,
      steps_per_epoch=100,
      epochs=ep,
      validation_data=validation_generator,
      validation_steps=50)
$\endgroup$
3
  • $\begingroup$ Sorry for the late response, I was on holidays.. Do you think using a pretrained network will help, even if my data looks nothing like real world images at all? I'm also probably going to try to use data augmentation to create more data. $\endgroup$ Jun 7, 2019 at 8:39
  • 1
    $\begingroup$ Thank you so much for the tips Peter! Only even using the pretrained model + the dense layers works like a charm! I guess I would improve it if you use the dataGen as well! $\endgroup$ Jun 7, 2019 at 12:15
  • $\begingroup$ good to hear that: happy coding! $\endgroup$
    – Peter
    Jun 7, 2019 at 14:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.