# Massive variation in results with tensorflow and keras

I'm new to Tensorflow and Keras and I some background knowledge of how CNN's work. I'm using a basic sequential model based on the code by https://pythonprogramming.net/convolutional-neural-network-deep-learning-python-tensorflow-keras/

I have a problem where my results variation is very big. The first time I ran the model today I got around 90% accuracy. But the runs after that were around 25% which is as good as guessing since I have four classes.

Here's my code:

tf.reset_default_graph()

batch_size = 32

logdir = os.path.join("logs", datetime.datetime.now().strftime("%Y%m%d-%H%M%S"))
tensorboard_callback = tf.keras.callbacks.TensorBoard(logdir, histogram_freq=1)

X = X/255.0

model = Sequential()

model.add(Flatten())  # this converts our 3D feature maps to 1D feature vectors

model.compile(loss='categorical_crossentropy',
metrics=['accuracy'])

with tf.Session() as sess:
model.fit(train_X, train_y, batch_size=batch_size, epochs=3,
validation_split=0.1, callbacks=[tensorboard_callback], validation_data=(test_X, test_y))


Am I doing something completely wrong with the model? I do have quite a small dataset, just 1640 images. But why do some runs perform so good then?

• your validation split is low (10%), this could be an issue. Did you fix the random state/validation set? It can be that you simply had a lucky shot with your val data... – Peter May 28 '19 at 13:41
• since you have a very small sample, you could also consider training on top of a pretrained model. – Peter May 28 '19 at 13:52
• I think I do sometimes get lucky shots.. as I'm speaking I get better results. Would it be better to increase the size of the validation set? What do you mean by fixing the state/validation set? I will look in to pretrained models (never really heard of it). – Bram Kreuger May 28 '19 at 14:36
• Thanks for the replies btw! :) – Bram Kreuger May 28 '19 at 14:37
• At the moment I'm reaching 100% accuracy, that doesn't seem good as well.. – Bram Kreuger May 28 '19 at 14:40

I commented previously, now some idea as an answer:

1) I think that the validation data may include "lucky shots" which match the trained patterns well in some cases. Also a small validation set can be a problem. As far as I know, validation data is the last X% of the data (in Keras). You can set shuffle=True to mix validation data or False to not mix it (as I understand). I would try this. https://github.com/keras-team/keras/issues/597

2) An option to get a good idea of how the model performs is cross validation. If you can afford it, train 3 or 5 or 10 models with different validation data and have a look at the average accuracy. Should give you also an idea of how volatile the results are.

3) You have a small sample. Thus, adding more validation data can be a problem. However, I would also try with 20%.

4) Since you have a small sample, the NN might have trouble getting all the relevant features. Instead of training a convnet from scratch, you can use a pretrained model. This has already learned a lot of features. Given a small sample, this could be THE way to go. Note that there are a number of different pretrained models which you can try. https://keras.io/applications/

5) Get more data (if possible).

I recently used a pretrained model in a multiclass setting. Here is my code, which is a slightly modified version of a tutorial code by F. Chollet (https://github.com/fchollet/deep-learning-with-python-notebooks/blob/master/5.3-using-a-pretrained-convnet.ipynb).

from keras.applications import VGG16
import os, datetime, statistics
import numpy as np
from keras.preprocessing.image import ImageDataGenerator
from keras.utils import to_categorical
from keras import models
from keras import layers
from keras import optimizers
from keras.layers.core import Dense, Dropout, Activation
from PIL import ImageFile

###############################################
# DIR with training images
base_dir = 'C:/kerasimages'
# Number training images
ntrain = 2000
# Number validation images
nval  = 500
# Batch size
batch_size = 20
# Epochs fine tuning
ep = 600
# Epochs first step
ep_first = 50
# Number of classes (for training, output layer)
nclasses = 22
###############################################

conv_base = VGG16(weights='imagenet', include_top=False, input_shape=(150, 150, 3))
train_dir = os.path.join(base_dir, 'train')
validation_dir = os.path.join(base_dir, 'val')

datagen = ImageDataGenerator(rescale=1./255)

def extract_features(directory, sample_count):
features = np.zeros(shape=(sample_count, 4, 4, 512))
labels = np.zeros(shape=(sample_count))
generator = datagen.flow_from_directory(
directory,
target_size=(150, 150),
batch_size=batch_size,
class_mode='binary')
i = 0
for inputs_batch, labels_batch in generator:
features_batch = conv_base.predict(inputs_batch)
features[i * batch_size : (i + 1) * batch_size] = features_batch
labels[i * batch_size : (i + 1) * batch_size] = labels_batch
i += 1
if i * batch_size >= sample_count:
break
return features, labels

# Lables and features
train_features, train_labels = extract_features(train_dir, ntrain)
validation_features, validation_labels = extract_features(validation_dir, nval)
train_labels = to_categorical(train_labels)
validation_labels = to_categorical(validation_labels)
train_features = np.reshape(train_features, (ntrain, 4 * 4 * 512))
validation_features = np.reshape(validation_features, (nval, 4 * 4 * 512))

#######################################
# Model
model = models.Sequential()
conv_base.trainable = False

#######################################
# Data generators
train_datagen = ImageDataGenerator(
rescale=1./255,
rotation_range=40,
width_shift_range=0.2,
height_shift_range=0.2,
shear_range=0.2,
zoom_range=0.2,
horizontal_flip=True,
fill_mode='nearest')

test_datagen = ImageDataGenerator(rescale=1./255)

train_generator = train_datagen.flow_from_directory(
train_dir,
target_size=(150, 150),
batch_size=batch_size,
class_mode='categorical')

validation_generator = test_datagen.flow_from_directory(
validation_dir,
target_size=(150, 150),
batch_size=batch_size,
class_mode='categorical')

# Model compile / fit
model.compile(loss='categorical_crossentropy',
optimizer=optimizers.RMSprop(lr=2e-5),
metrics=['acc'])

history = model.fit_generator(
train_generator,
steps_per_epoch=100,
epochs=ep_first,
validation_data=validation_generator,
validation_steps=50,
verbose=2)

#######################################
# Fine tuning
conv_base.trainable = True

set_trainable = False
for layer in conv_base.layers:
if layer.name == 'block5_conv1':
set_trainable = True
if set_trainable:
layer.trainable = True
else:
layer.trainable = False

model.compile(loss='categorical_crossentropy',
optimizer=optimizers.RMSprop(lr=1e-5),
metrics=['acc'])

history = model.fit_generator(
train_generator,
steps_per_epoch=100,
epochs=ep,
validation_data=validation_generator,
validation_steps=50)

• Sorry for the late response, I was on holidays.. Do you think using a pretrained network will help, even if my data looks nothing like real world images at all? I'm also probably going to try to use data augmentation to create more data. – Bram Kreuger Jun 7 '19 at 8:39
• Thank you so much for the tips Peter! Only even using the pretrained model + the dense layers works like a charm! I guess I would improve it if you use the dataGen as well! – Bram Kreuger Jun 7 '19 at 12:15
• good to hear that: happy coding! – Peter Jun 7 '19 at 14:53