# CNN for classification giving extreme result probabilities

I'm having issues with my CNN, using Keras with Theano backend. Basically, I need to classify 340x340 grayscale images into 6 categories. The problem is my CNN gives too "hard" probabilities, for instance it will rarely give predictions with some uncertainty, and always tries to push for a 90%+ for one class. The problem is that for my coursework, the penalty used is very harsh for complete miss classification, and uncertainty is much preferred. ( so having a prediction like [0.6, 0.3, 0.2, ...] is much better than having [0.9,0.03,0.02,..].

I'm unsure why this is happening. My dataset consists of 2400 images, which are from different CCTV, and task is about recognising possible objects. Only 800 of the samples are actually from the data, the other 1600 have been generated through data augmentation. Note that it is therefore extremely likely a that some pictures are either identical, or extremely similar (e.g. the same scene, one second later)

model = Sequential()
#1
# Few filter to take big stuff out
# Also, first layer is not conv so that I can reuse that layer separately
model.add(Dropout(0.1, input_shape=(1,340,340)))
model.add(Convolution2D(64, 4, 4, border_mode='same'))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2), dim_ordering="th"))

#2
model.add(Dropout(0.1))
model.add(Convolution2D(128, 4, 4, border_mode='same'))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2), dim_ordering="th"))

#3
model.add(Dropout(0.1))
model.add(Convolution2D(256, 4, 4, border_mode='same'))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2), dim_ordering="th"))

#4
model.add(Flatten())
model.add(Dense(1024))
model.add(Activation('relu'))
#5
model.add(Dense(512))
model.add(Activation('relu'))
#6
model.add(Dense(6))
model.add(Activation('softmax'))

opt = SGD(lr=0.001)
model.compile(loss='categorical_crossentropy', optimizer=opt, metrics=['accuracy'])

print "Training.."

filepath = "log/weights-improvement-{epoch:02d}---{val_acc:.2f}.hdf5"
checkpoint = ModelCheckpoint(filepath, monitor='val_acc', verbose=1, save_best_only=True, mode='max')
callbacks_list = [checkpoint]
model.fit(X_t, y_t, validation_split=0.1, nb_epoch=500, batch_size=32, callbacks=callbacks_list)


How do you suggest I fix this? Thank you in advance!

• Could you clarify how the model is being trained, cross-validated and tested? Especially how you split the data. Do the images fall into natural groups separate from the classes - e.g. if this is facial expression classes, are there several images the same person with different expressions? That is likely to be relevant. Your model code looks OK, except perhaps you should not really care about accuracy metric, just use the same metric as your course is using - that won't affect the results directly, but should make it quicker to pick the best performing model. – Neil Slater Mar 13 '17 at 12:48
• @NeilSlater Thank you for answering, I've updated the question adding information about the images and the lines of code in charge of training, with 0.1 validation split. – Dominus Mar 13 '17 at 16:30
• Which scores are bad? Training loss, validation loss, or test loss? If test loss, how is testing being done - do the course instructors have a test set separate to the data you have been provided? – Neil Slater Mar 13 '17 at 16:53
• @Dominus Pls elaborate on this penalty on misclassified instances. Have you thought of incorporating this penalty into your loss? – horaceT Mar 13 '17 at 17:16
• @NeilSlater Training and validation both seem to be reliably good - above 60%. Then, the testing is done on a dataset (not classified obviously) but publicly available, and the pictures look pretty much the same as the ones provided for training. The results of the testing are then saved in a csv, uploaded to a server which then returns the final score. – Dominus Mar 13 '17 at 18:38

## 1 Answer

There are a few different factors involved here. It is difficult to tell, without getting heavily involved, which could be the most important. I will put them in the order I think worth looking at first.

• Your data is images from CCTV, so you likely have more than one image from each camera. From your results (reasonable training and CV scores, but bad training), it looks like you are over-fitting. But your CV approach is not spotting this. So I think that the test set is likely to be from a different set of cameras to the training set. In order to properly measure CV therefore, you have to split train/cv by camera - you cannot just use the 0.1 split, because a random split will include images which are correlated with training data and will give you too high an estimate, allowing you to overfit without noticing.

• It occurs to me it might just be your data augmentation causing a problem for you here. If you augment first then randomly split to train/CV, then your CV set will contain images very similar to training set, and will get too high a score. You can more easily check and fix this than split by camera, so give it a try.
• 800 original images is not much to work with. You need to do something about that. Here are some ideas:

• Scale down the images. You probably don't need 340x340. Depending on the target object, maybe just 78x78 will do. You can assess this easily enough - scale down and check if you can still differentiate the classes easily by eye.

• You don't have enough data to get best quality filters in the lower layers, which will limit the capabilities of the CNN. You might bootstrap from a pre-trained image model. Take a publicly available pre-trained CNN, such as VGG-19, use its weights in convolutional layers as a starting point, put different classifier layers on top and fine-tune your classifier starting from this. This might change the ideal image scale too - you want something that fits the pre-trained CNN.

• Augmenting the data, which you have started. You could go a lot further. Take random patches from training examples, possibly flipped horizontally (if this maintains the object class). However, don't augment data used for cross validation, unless your model used for testing also includes augmentation - e.g. if it takes 8 random augmented variants of the test image and returns an average of predictions, then you can do similar for cross-validation.

• A 0.1 train/CV split run once on this much data is not going to give you an accurate assessment of the model. You need to run k-fold cross-validation. This is annoying because NNs take a long time to train, but if you want some confidence that you have really found some good parameters, you will need to do this. Remember to split by camera if you can.

• The scoring appears to be strongly related to categorical cross-entropy, so you have the right loss function. You should optimise, using cross-validation, to find the model with the lowest loss, not the best accuracy.

• Thank you Neil! A few questions about your answer: - I'm not confident with what you said about bootstrapping a pre-trained model, do you have any good ressource regarding that step? - When you say don't augment data used for cross-validation, do you mean I should test the data CNN only on the original data, but train it with the augmented? - How do you implement K-fold validation on the CNN as shown? I looked for it, but the only pre-built option I found is the one I'm using, with a single fold. Do you have any code/ressource for that? Thank you in advance! – Dominus Mar 13 '17 at 20:08
• @Dominus: I don't have resources for all my suggestions, you will need to research further for most of them. However, for bootstrapping from existing models in Keras: keras.io/applications "do you mean I should test the data CNN only on the original data, but train it with the augmented?" Yes, that's exactly what I mean. Otherwise you take the risk that you are writing a classifier for augmented data only. However, if you can augment the test set too for prediction, then you can just augment everything . . . – Neil Slater Mar 13 '17 at 20:12
• Thank you again, I have a last small question, just answer instinctively, do you think the overall CNN structure is good? or maybe should I get rid of the last CNN layer or add Dense layers at the end? – Dominus Mar 13 '17 at 20:16
• @Dominus: The CNN design looks OK to me, I cannot see anything wrong with it. If you keep the larger input image size, you might want to add a convolutional layer or two, but I'm just guessing. – Neil Slater Mar 13 '17 at 20:21