# Interpreting confusion matrix and validation results in convolutional networks

I need some help in the assessment of the training results of a convolutional neural network. Here is my setup:

• Architecture: InceptionV3
• Pre-trained InceptionV3 with weights from image net
• replaced last layer, retrained the second half of the network
• Classification: Softmax with cross entropy ploss
• Adam optimizer with keras default parameters
• Hyper parameters: Learning rate 0.0001, batch size 64
• Goal: Multi-label classification with exclusive labels
• Data: 17 classes ranging from 0-16 with 48.600 images in total
• Train, Test, Validation: 80%, 10%, 10%
• Training: 2 epochs training only last layer + 2 epochs training second half of network

Results

Based on my validation set I created a confusion matrix with the following results.

Here are also some more per-class performance metrics:

As you can see, performance is not very good. The test-accuracy was at 62.2% so it was also not very good. Looking at the confusion matrix it is noticeable that the classes 0, 2, 3, 7, 10, 11, 13 and 14 are never predicted. These classes have also the smallest number of samples. I wonder why this happens? Those classes are more rare than the others but nver predicting them seems odd. What would you recommend going forward?

• Is the training data balanced (i.e. same ratio of individuals for each class)? – ncasas Mar 11 '18 at 9:40
• looking into the same direction like @ncasas, did you try to rotate the assignment of samples between training, test and validation? – Frankstr Mar 11 '18 at 9:42
• @ncasas: No, all the data sets (train/test/validate) have the same distribution of data that can be found "in the wild". What are your experiences with this? I did some research and found that it is probably not wise to artificially smoothen the distribution between classes. – Gegenwind Mar 11 '18 at 10:03
• @Frankstr: What do you mean by rotate? I strictly split between train/test and dev sets in order to avoid leaking information about latter datasets into the training. – Gegenwind Mar 11 '18 at 10:03
• I stand corrected. This recent paper (arxiv.org/pdf/1710.05381.pdf) investigates the effect of imbalance and finds a large negative effect, also influenced by the number of classes that are out of balance. It also suggests oversampling (randomly add samples from under represented classes to achieve class balance). – Gegenwind Mar 11 '18 at 10:47

• Because your data-set is not balanced, use other metrics to evaluate your model, like F1 score.
• There are different reasons for performing poorly on some classes. I don't know whether you are capable to change the model or not, but that is first thing that can be done. Your pre-trained model has a lot of parameters to be trained. The main reason it performs bad is the low number of training epochs. The loss function is ok, you have to specify class weights to care about classes with rare data more. F1 is better than accuracy because your data-set is unbalanced. As you can see most entries for F1 score has zero value which means you are performing bad on your data. – Media Mar 11 '18 at 10:14