2
$\begingroup$

I need some help in the assessment of the training results of a convolutional neural network. Here is my setup:

  • Architecture: InceptionV3
  • Pre-trained InceptionV3 with weights from image net
  • replaced last layer, retrained the second half of the network
  • Classification: Softmax with cross entropy ploss
  • Adam optimizer with keras default parameters
  • Hyper parameters: Learning rate 0.0001, batch size 64
  • Goal: Multi-label classification with exclusive labels
  • Data: 17 classes ranging from 0-16 with 48.600 images in total
  • Train, Test, Validation: 80%, 10%, 10%
  • Training: 2 epochs training only last layer + 2 epochs training second half of network

Results

Based on my validation set I created a confusion matrix with the following results.

enter image description here

Here are also some more per-class performance metrics:

enter image description here

As you can see, performance is not very good. The test-accuracy was at 62.2% so it was also not very good. Looking at the confusion matrix it is noticeable that the classes 0, 2, 3, 7, 10, 11, 13 and 14 are never predicted. These classes have also the smallest number of samples. I wonder why this happens? Those classes are more rare than the others but nver predicting them seems odd. What would you recommend going forward?

Improve this question
$\endgroup$
5
  • $\begingroup$ Is the training data balanced (i.e. same ratio of individuals for each class)? $\endgroup$
    – noe
    Mar 11, 2018 at 9:40
  • $\begingroup$ looking into the same direction like @ncasas, did you try to rotate the assignment of samples between training, test and validation? $\endgroup$
    – Frankstr
    Mar 11, 2018 at 9:42
  • $\begingroup$ @ncasas: No, all the data sets (train/test/validate) have the same distribution of data that can be found "in the wild". What are your experiences with this? I did some research and found that it is probably not wise to artificially smoothen the distribution between classes. $\endgroup$
    – Gegenwind
    Mar 11, 2018 at 10:03
  • $\begingroup$ @Frankstr: What do you mean by rotate? I strictly split between train/test and dev sets in order to avoid leaking information about latter datasets into the training. $\endgroup$
    – Gegenwind
    Mar 11, 2018 at 10:03
  • $\begingroup$ I stand corrected. This recent paper (arxiv.org/pdf/1710.05381.pdf) investigates the effect of imbalance and finds a large negative effect, also influenced by the number of classes that are out of balance. It also suggests oversampling (randomly add samples from under represented classes to achieve class balance). $\endgroup$
    – Gegenwind
    Mar 11, 2018 at 10:47

1 Answer 1

Reset to default
1
$\begingroup$

These are my recommendations:

  • Try to train your model more, with different learning rates. Your current learning rate is so much small. At first steps put it to a big value then reduce it smoothly.
  • Try to change the architecture. If your task does not have the same scale as image-net surely you would need a network with simpler network architecture, consequently the number of parameters diminishes.
  • Changing the distribution of data is dangerous, but you can augment your entire data-set by some extent to let your network see different and more examples.
  • For those classes that have not been learnt you can set the class-weight to a bigger number to construct a cost function which cares more about those classes.
  • Because your data-set is not balanced, use other metrics to evaluate your model, like F1 score.
Improve this answer
$\endgroup$
3
  • $\begingroup$ Thanks for your suggestions. Can you provide some reasoning why some classes underperform? Regarding "using other metrics": Do you suggest to replace the loss function? Currently I use cross entropy loss for that, which I thought would be fitting. For my own evaluation you can see that F1-score is among the performance metrics. However I am not sure which actions it implies. $\endgroup$
    – Gegenwind
    Mar 11, 2018 at 10:06
  • $\begingroup$ There are different reasons for performing poorly on some classes. I don't know whether you are capable to change the model or not, but that is first thing that can be done. Your pre-trained model has a lot of parameters to be trained. The main reason it performs bad is the low number of training epochs. The loss function is ok, you have to specify class weights to care about classes with rare data more. F1 is better than accuracy because your data-set is unbalanced. As you can see most entries for F1 score has zero value which means you are performing bad on your data. $\endgroup$
    – Green Falcon
    Mar 11, 2018 at 10:14
  • 1
    $\begingroup$ Thanks for your suggestions media. I'll try them out and report back if the helped solving the issue. $\endgroup$
    – Gegenwind
    Mar 13, 2018 at 7:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.