Loss is bad, but accuracy increases?

Question

I have a multicategorial classification problem for images. There are 5 (imbalanced) classes for which i use different class weights. In general there are only a few training images per class: ~56-238

To classify them, I use a neural network with much data augmentation. I have a validation set with the same distribution as the train set (but it only has about 30% of the images per class).

The resulting loss / accuracy graphs look a bit weird (edit: the second graph contains the term "Test Loss", but it is the "Validation Loss"):

I'm not sure how I can interpret these two images: The validation accuracy cleary increases, but the validation loss doesn't change too much. Can anyone help me with the interpretetion of these graphs?

Thank you very much

Answer 1

If you have an imbalanced dataset, you may want to take steps to resample. When evaluating your results of your algorithm, it may be a good idea to choose a metric that is better suited for class imbalance problems.

Accuracy is not a good metric for evaluating the performance of class imbalance problems because accuracy is rewarded for predicting the most commonly occurring label. Accuracy is the measurement of the the number of labels predicted correctly over the number of all the observations. For example, if you had 99% of the data with label 'a' and 1% of the data with label 'b', and you then labeled the entire dataset with label 'a', your accuracy would be 99%. However, your ability to label 'b' correctly was 0%. Log loss is also not particularly good for class imbalance problems. You may want to consider weighted log loss instead.

Here are some other metrics that you can investigate.

Precision and Recall

Precision is the ratio of correctly predicted positive labels to the total predicted positive observations. Recall is the ratio of correctly predicted positive observations to the all observations in the given class. You can read more about these metrics here. For a Binary Classification problem, you can view the predicted class vs the actual class in a confusion matrix.

F1 Score

F1 Score is the weighted average of precision and recall. F1 score takes false negatives and false positives into account. So, it is more informative than accuracy when evaluating class imbalance problems.

F1 Score exists as a Python package in sklearn. It also exists as a function in R.

Area Under Precision Recall Curve

You can also investigate Area under precision recall curve (PR), which the measurement under the area of precision recall curve plot. It can be used to evaluate large class imbalance problems. PR Curve exists as a python package in sklearn and an R package.