I am using Keras/CNN to identify plankton images collected with an in situ camera. When making confusion matrices on the test sets following training I am finding that the accuracy from the predictions is quite poor.
I have a large number of files and have been using flow_from_directory and generators, I suspect that there might be something happening with the indexing of the predictions (e.g. this post), but as near as I can tell the indexing of the filenames/labels is matching up.
I worked up a quick example that is similar to what I am doing with the mnist_png dataset:
import numpy as np from keras.models import Sequential from keras.layers import Conv2D, MaxPooling2D from keras.layers import Activation, Dropout, Flatten, Dense from keras.layers.advanced_activations import LeakyReLU from keras import backend as K from keras.preprocessing.image import ImageDataGenerator img_width, img_height = 28, 28 train_data_dir = 'S:/mnist_png/training' num_epochs = 100 batch_size = 128 num_test_samples=10000 if K.image_data_format() == 'channels_first': input_shape = (3, img_width, img_height) else: input_shape = (img_width, img_height, 3) model = Sequential() model.add(Conv2D(32, (3, 3), input_shape=input_shape)) model.add(Activation('relu')) model.add(Conv2D(16, (3, 3))) model.add(Activation('relu')) model.add(MaxPooling2D(pool_size=(3, 3))) model.add(Dropout(0.5)) model.add(Conv2D(64, (3, 3))) model.add(Activation('relu')) model.add(Conv2D(32, (3, 3))) model.add(Activation('relu')) model.add(MaxPooling2D(pool_size=(3, 3))) model.add(Dropout(0.5)) model.add(Flatten()) model.add(Dense(512,activation='linear')) model.add(LeakyReLU(alpha=.3)) model.add(Dropout(0.5)) model.add(Dense(512,activation='linear')) model.add(LeakyReLU(alpha=.3)) model.add(Dense(10)) model.add(Activation('softmax')) model.compile(loss='categorical_crossentropy', optimizer='Adam', metrics=['accuracy']) # augmentation for training train_datagen = ImageDataGenerator( rescale=1. / 255, shear_range=0.2, zoom_range=0.2, horizontal_flip=True, vertical_flip=True, rotation_range=90, validation_split=0.1) train_generator = train_datagen.flow_from_directory( train_data_dir, target_size=(img_width, img_height), batch_size=batch_size, class_mode='categorical') model.fit_generator( train_generator, steps_per_epoch=batch_size, epochs=num_epochs)
...after 100 epochs I'm getting loss: 0.7517 - acc: 0.7482.
I then evaluate the test set thusly:
test_data_dir = 'S:/mnist_png/testing' test_datagen = ImageDataGenerator( rescale=1. / 255) test_generator = test_datagen.flow_from_directory( test_data_dir, target_size=(img_width, img_height), batch_size=batch_size, shuffle='False', class_mode='categorical') #Evaluate model on test set scores = model.evaluate_generator(test_generator,workers=12)
...the scores for that were 0.6184 and 0.8168, so in the same ballpark.
But it gets weird when I look at the predictions, e.g.:
test_generator.reset()# Necessary to force it to start from beginning Y_pred = model.predict_generator(test_generator) y_pred = np.argmax(Y_pred, axis=-1) sum(y_pred==test_generator.classes)/10000
The proportion of predictions that are actually correct (calculated in the last line) is around 0.1; when I look at a confusion matrix it's all over the place and the diagonal is a lot of zeros.
I have verified that the test_generator.classes match up with the directories in test_generator.filenames, and shuffle is off. Per this post calling test_generator.reset() should force it to take the files in order, but I'm wondering if it is not.
Does anyone have any thoughts on why this is happening or further steps to troubleshoot it?