2
$\begingroup$

I'm working on a Classification problem as a side project and I'm receiving results contrary to what I'd expect. With 100,000 records, each with 7 components for X, the model is performing much better with 70% of the data being used to test, rather than what I'd expect: 70% training split to work better.

Has anyone had this before or know why this could be? I'm wondering if maybe the large size of the data is worsening the model somehow.

$\endgroup$
1
  • $\begingroup$ have you done the fine tuning in the two cases that you have. Note that each time you have different input data, the parameters need to be changed accordingly unless a thorough investigation is done to find general parameters which could work for both scenarios. $\endgroup$
    – prashanth
    May 22, 2023 at 7:40

2 Answers 2

1
$\begingroup$

Is this data is imbalanced, like 95% target A versus 5% target B? If it is I would suggest that the test set sample was a poor representation of the under represented target to be classified. Could you augment the data set to increase its size, e.g. if it's a time-series use other data points, image recognition rotate or shift the hues, contrast, orientation? Dealing imbalance has alternative solutions if thats the issue.


From the comments: The issue is 92% for 30:70% train-test split and 80% for 70:30% train-test split.

You could simply say 80% is good enough I'll proceed with the orthodox 70:30 split. If you are proceeding with 30:70 split you would need to be clear about that, if it's a manuscript the reviewer would likely return it. Personally, I don't think it's cool.

I get the impression that 3 of the targets under classification have approximately equal proportions (just guessing). The issue is whether there is a minority part of the classification which is getting misrepresented in the testing split.

There are two approaches I would use (as a data scientist):

  1. Reduce the problem to the 3 majority categories and see if the discrepancy continues between 30:70 and 70:30
  2. Augment the data and use a standard 70:30 split, however now the 30% is more like the original 70% due to augmentation.

My suspicion is in point 1 is the discrepancy will disappear, thus you've identified the problem and can consider whether its worth moving to point 2.

If that was correct it's what does the fourth catagory represent and how important this is to you. For example in cancer that 4th category (the smallest) could be really important because it carries the highest mortality. If it's just not important - its a minority variant that no-one cares about and you just state the classification for this category needs further development (which might never happen).

Its area specific. In my problems I can't discount a minority classification, but thats because it might become the variant that takes over the world and I've just missed it (I do evolutionary selection). In your problem and I get the impression in many business related analytics you can.

$\endgroup$
5
  • $\begingroup$ In the case of this data there are 4 potential classifications, with the most common occuring ~36% of the time, so it is somewhat imbalanced but not enormously so - I'll research what i can do regarding augmenting the data $\endgroup$ May 16, 2023 at 16:22
  • $\begingroup$ Thanks @GroupTheory14 if you've a target at <= 2% (possible) then it's a reasonable explanation. Depends what the frequency of the least frequent target. The imbalance will need to be large, otherwise I'd look at other factors. $\endgroup$
    – M__
    May 16, 2023 at 16:38
  • 1
    $\begingroup$ Thanks for the help - Currently my model is running at 92% on the train (30% data) and 80% on the test (70%) data. Doing the split/hyperparameters this way was the highest accuracy I could get on the test data. Is this necessarily a problem? For my task, 80% is definitely a good enough accuracy, I just wondered if it perhaps indicates a large flaw in the model which may come back to bite me later - I know its hard to say without much info but do you think this could be the case? $\endgroup$ May 16, 2023 at 17:29
  • $\begingroup$ @GroupTheory14 I've updated my response below the line. $\endgroup$
    – M__
    May 16, 2023 at 17:57
  • 1
    $\begingroup$ @M_ Thanks so much for the help, given me a lot of things to think about and implement! I'll do some playing around at work tomorrow. $\endgroup$ May 16, 2023 at 18:43
2
$\begingroup$

The results you are receving may be affected by variance.

When you evaluate the model on the 30% of the data, you will have low bias but more variance.

The imbalance of the target should not be a problem as long as you stratify your split.

Alternatives to consider:

  1. Use the out of bag score of random forest, that is the score in the samples that are excluded during bootstrap. That will give you a good approximation of the performance on the test and they will be approximately 30% of the train data always so it will be more fair evaluation.

  2. Create learning curves

Evaluate the model performance as a function of sample size, so you can see if the model actually benefits from adding more data to the training set.

Hope it helps!

$\endgroup$
1
  • $\begingroup$ Thank you! Definitely gives me some options to look into, I'll give them a go at work tomorrow and see how they go :) Thanks again $\endgroup$ May 16, 2023 at 18:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.