CNN for classification giving extreme result probabilities

Question

I'm having issues with my CNN, using Keras with Theano backend. Basically, I need to classify 340x340 grayscale images into 6 categories. The problem is my CNN gives too "hard" probabilities, for instance it will rarely give predictions with some uncertainty, and always tries to push for a 90%+ for one class. The problem is that for my coursework, the penalty used is very harsh for complete miss classification, and uncertainty is much preferred. ( so having a prediction like [0.6, 0.3, 0.2, ...] is much better than having [0.9,0.03,0.02,..].

I'm unsure why this is happening. My dataset consists of 2400 images, which are from different CCTV, and task is about recognising possible objects. Only 800 of the samples are actually from the data, the other 1600 have been generated through data augmentation. Note that it is therefore extremely likely a that some pictures are either identical, or extremely similar (e.g. the same scene, one second later)

model = Sequential()
#1 
# Few filter to take big stuff out
# Also, first layer is not conv so that I can reuse that layer separately
model.add(Dropout(0.1, input_shape=(1,340,340)))
model.add(Convolution2D(64, 4, 4, border_mode='same'))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2), dim_ordering="th"))   

#2
model.add(Dropout(0.1))
model.add(Convolution2D(128, 4, 4, border_mode='same'))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2), dim_ordering="th"))   

#3
model.add(Dropout(0.1))
model.add(Convolution2D(256, 4, 4, border_mode='same'))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2), dim_ordering="th"))   

#4
model.add(Flatten())
model.add(Dense(1024))
model.add(Activation('relu'))
#5
model.add(Dense(512))
model.add(Activation('relu'))
#6
model.add(Dense(6))
model.add(Activation('softmax'))

opt = SGD(lr=0.001)
model.compile(loss='categorical_crossentropy', optimizer=opt, metrics=['accuracy'])

print "Training.."


filepath = "log/weights-improvement-{epoch:02d}---{val_acc:.2f}.hdf5"
checkpoint = ModelCheckpoint(filepath, monitor='val_acc', verbose=1, save_best_only=True, mode='max')
callbacks_list = [checkpoint] 
model.fit(X_t, y_t, validation_split=0.1, nb_epoch=500, batch_size=32, callbacks=callbacks_list)

How do you suggest I fix this? Thank you in advance!

Answer 1

There are a few different factors involved here. It is difficult to tell, without getting heavily involved, which could be the most important. I will put them in the order I think worth looking at first.

• Your data is images from CCTV, so you likely have more than one image from each camera. From your results (reasonable training and CV scores, but bad training), it looks like you are over-fitting. But your CV approach is not spotting this. So I think that the test set is likely to be from a different set of cameras to the training set. In order to properly measure CV therefore, you have to split train/cv by camera - you cannot just use the 0.1 split, because a random split will include images which are correlated with training data and will give you too high an estimate, allowing you to overfit without noticing.

  • It occurs to me it might just be your data augmentation causing a problem for you here. If you augment first then randomly split to train/CV, then your CV set will contain images very similar to training set, and will get too high a score. You can more easily check and fix this than split by camera, so give it a try.

• 800 original images is not much to work with. You need to do something about that. Here are some ideas:

  • Scale down the images. You probably don't need 340x340. Depending on the target object, maybe just 78x78 will do. You can assess this easily enough - scale down and check if you can still differentiate the classes easily by eye.

  • You don't have enough data to get best quality filters in the lower layers, which will limit the capabilities of the CNN. You might bootstrap from a pre-trained image model. Take a publicly available pre-trained CNN, such as VGG-19, use its weights in convolutional layers as a starting point, put different classifier layers on top and fine-tune your classifier starting from this. This might change the ideal image scale too - you want something that fits the pre-trained CNN.

  • Augmenting the data, which you have started. You could go a lot further. Take random patches from training examples, possibly flipped horizontally (if this maintains the object class). However, don't augment data used for cross validation, unless your model used for testing also includes augmentation - e.g. if it takes 8 random augmented variants of the test image and returns an average of predictions, then you can do similar for cross-validation.

• A 0.1 train/CV split run once on this much data is not going to give you an accurate assessment of the model. You need to run k-fold cross-validation. This is annoying because NNs take a long time to train, but if you want some confidence that you have really found some good parameters, you will need to do this. Remember to split by camera if you can.

• The scoring appears to be strongly related to categorical cross-entropy, so you have the right loss function. You should optimise, using cross-validation, to find the model with the lowest loss, not the best accuracy.