Is this data is imbalanced, like 95% target A versus 5% target B? If it is I would suggest that the test set sample was a poor representation of the under represented target to be classified. Could you augment the data set to increase its size, e.g. if it's a time-series use other data points, image recognition rotate or shift the hues, contrast, orientation? Dealing imbalance has alternative solutions if thats the issue.
From the comments:
The issue is 92% for 30:70% train-test split and 80% for 70:30% train-test split.
You could simply say 80% is good enough I'll proceed with the orthodox 70:30 split. If you are proceeding with 30:70 split you would need to be clear about that, if it's a manuscript the reviewer would likely return it. Personally, I don't think it's cool.
I get the impression that 3 of the targets under classification have approximately equal proportions (just guessing). The issue is whether there is a minority part of the classification which is getting misrepresented in the testing split.
There are two approaches I would use (as a data scientist):
- Reduce the problem to the 3 majority categories and see if the discrepancy continues between 30:70 and 70:30
- Augment the data and use a standard 70:30 split, however now the 30% is more like the original 70% due to augmentation.
My suspicion is in point 1 is the discrepancy will disappear, thus you've identified the problem and can consider whether its worth moving to point 2.
If that was correct it's what does the fourth catagory represent and how important this is to you. For example in cancer that 4th category (the smallest) could be really important because it carries the highest mortality. If it's just not important - its a minority variant that no-one cares about and you just state the classification for this category needs further development (which might never happen).
Its area specific. In my problems I can't discount a minority classification, but thats because it might become the variant that takes over the world and I've just missed it (I do evolutionary selection). In your problem and I get the impression in many business related analytics you can.