I built a CNN image classifier for a dataset that contains 6 classes. The dataset is balanced in all 6 classes. After training, the model gives pretty good prediction accuracy in all but 2 classes. To elaborate further, let us label these 6 classes with integers from '0' to '5'. The trained model does well in predicting classes from '0' to '3'. But almost 5%-10% of class '4' image is predicted as class '5' and similarly, 5%-10% of class '5' image is predicted as class '4'.
How should I deal with this situation?