0
$\begingroup$

I use a MLP to classify three different classes A, B, C. The loss function I use is categorical cross entropy and the optimiser is adam. To estimate my models performance I use 10-fold Cross Validation. On average i get 60% accuracy score but I need it to be higher. The confusion matrix for the classes A,B,C, I get is the following:

Class A Class B Class C
14440 8118 11229
6045 21863 5879
6207 4264 23315

The amount of data points I have for each class is sufficiently large and equal (the ratio is 1:1:1). I was trying to see how the model would fare, if instead of 3 classes it classified two classes. Each time I remove from the dataset the data points of the class I wont be using. So when I train it to classify data points between class B and C I get around 80% accuracy scoreand the following confusion matrix:

Class B Class C
26456 7331
6255 27531

However when I train the model to classify data points between A and C i get around 69.5% and the following confusion matrix:

Class A Class C
22180 11607
9659 24127

For the classes A and B I get around 72% accuracy score and the following confusion matrix:

Class A Class B
23971 9816
9616 24170

In all cases the precision score, recall score and f1 score are more or less equal to the accuracy score.

  • Could the reason why I get low accuracy when I classify the 3 classes be class A?
  • Maybe some data points from class A are too similar with data points from class B and others are too similar with data points of class C?

    If so what could I do to improve the score provided that I can't improve the dataset.
Improve this question
$\endgroup$
3
  • $\begingroup$ It's true that class A seems to be slightly harder to identify than B or C, but it's not that big a difference. It's clear that the problem becomes a bit harder with 3 classes, since you obtain 60% instead of at least 69% in all the binary cases (this is expected). Did you check that there is no overfitting? You can also try other models, but if the features are not sufficiently good indicators there's not much you can do about it. $\endgroup$
    – Erwan
    Commented Oct 18, 2021 at 15:49
  • $\begingroup$ @Erwan Doesnt 10-fold cross validation reduce overfitting? (at least to some extent) I will see how other models perform too. Thanks for your comment! $\endgroup$
    – Insomniac
    Commented Oct 18, 2021 at 16:02
  • $\begingroup$ No, CV doesn't reduce overfitting, it's only useful to obtain a reliable estimate of the performance. You should check overfitting by measuring performance on the training set: if it's much higher than on the test set, then there is overfitting. $\endgroup$
    – Erwan
    Commented Oct 18, 2021 at 16:20

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.