I use a MLP to classify three different classes A, B, C. The `loss function` i use is `categorical cross entropy` and the optimiser is `adam`. To estimate my models performance i use 10-fold Cross Validation. On average i get `60% accuracy score` but i need it to be higher. The confusion matrix for the classes A,B,C i get is the following: <br>

| Class A | Class B  | Class C  |
|:---- |:------:| -----:|
| 14440  | 8118    | 11229 |
| 6045  | 21863    | 5879 |
| 6207  | 4264    | 23315 |

The amount of data points i have for each class is sufficiently large and equal (the ratio is 1:1:1). I was trying to see how the model would fare, if instead of 3 classes it classified two classes. Each time i remove from the dataset the data points of the class i wont be using. So when i train it to classify data points between class B and C i get around `80% accuracy score`and the following confusion matrix:

Class B | Class C  |
|:---- |:------:| 
| 26456  | 7331    |
| 6255  | 27531    |

 However when i train the model to classify data points between A and C i get around `69.5%` and the following confusion matrix:

Class A | Class C  |
|:---- |:------:| 
| 22180  | 11607    |
| 9659  | 24127    |

 For the classes A and B i get around `72% accuracy score` and the following confusion matrix:

| Class A | Class B  |
|:---- |:------:|
| 23971  | 9816    |
| 9616  | 24170    |
 
In all cases the `precision score`, `recall score` and `f1 score` are more or less equal to the accuracy score. Could the reason why i get low accuracy when i classify the 3 classes be class A? Maybe some data points from class A are too similar with data points from class B and others are too similar with data points of class C? If so what could i do to improve the score provided that i can't improve the dataset.  
Thanks in advance!