Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
I'm working on a Classification problem as a side project and I'm receiving results contrary to what I'd expect. With 100,000 records, each with 7 components for X, the model is performing much better with 70% of the data being used to test, rather than what I'd expect: 70% training split to work better.
Has anyone had this before or know why this could be? I'm wondering if maybe the large size of the data is worsening the model somehow.
Is this data is imbalanced, like 95% target A versus 5% target B? If it is I would suggest that the test set sample was a poor representation of the under represented target to be classified. Could you augment the data set to increase its size, e.g. if it's a time-series use other data points, image recognition rotate or shift the hues, contrast, orientation? Dealing imbalance has alternative solutions if thats the issue.
From the comments: The issue is 92% for 30:70% train-test split and 80% for 70:30% train-test split.
You could simply say 80% is good enough I'll proceed with the orthodox 70:30 split. If you are proceeding with 30:70 split you would need to be clear about that, if it's a manuscript the reviewer would likely return it. Personally, I don't think it's cool.
I get the impression that 3 of the targets under classification have approximately equal proportions (just guessing). The issue is whether there is a minority part of the classification which is getting misrepresented in the testing split.
There are two approaches I would use (as a data scientist):
My suspicion is in point 1 is the discrepancy will disappear, thus you've identified the problem and can consider whether its worth moving to point 2.
If that was correct it's what does the fourth catagory represent and how important this is to you. For example in cancer that 4th category (the smallest) could be really important because it carries the highest mortality. If it's just not important - its a minority variant that no-one cares about and you just state the classification for this category needs further development (which might never happen).
Its area specific. In my problems I can't discount a minority classification, but thats because it might become the variant that takes over the world and I've just missed it (I do evolutionary selection). In your problem and I get the impression in many business related analytics you can.
The results you are receving may be affected by variance.
When you evaluate the model on the 30% of the data, you will have low bias but more variance.
The imbalance of the target should not be a problem as long as you stratify your split.
Alternatives to consider:
Use the out of bag score of random forest, that is the score in the samples that are excluded during bootstrap. That will give you a good approximation of the performance on the test and they will be approximately 30% of the train data always so it will be more fair evaluation.
Create learning curves
Evaluate the model performance as a function of sample size, so you can see if the model actually benefits from adding more data to the training set.
Hope it helps!
