I fit my dataset to the random forest classifier and found that the model performance would vary among different sets of train and test data split. As what I have observed, it would jump from 0.67 to 0.75 in AUC under ROC curve (fitted by the same model under same setting of parameters) and the underlying range may be wider than that. So what is the issue behind this phenomena and how to deal with this problem? As my understanding, cross validation is used for a specific split of train and test data.
While training, your model will not have the same output when you train with different parts of the dataset. Cross validation is used to help negate this, by rotating the training and validation sets and training more.
Your dataset most likely has high variance, given the large jump in accuracy based on different validation sets. This means that the data is spread out, and can result in overfitting the model. You can imagine an overfitted model like this:
The green line represents the overfitted model.
Common techniques to reduce overfitting in random forests is k-fold Cross Validation, with k being between 5 and 10, and growing a larger forest.
What you are experiencing is not a problem, but rather an inherent attribute of all classifiers. The performance of a classifier depends on the training set, therefore the performance will vary with different training sets.
To find the best parameters for a specific classifier you will therefore want to vary training and test split (such as in crossvalidation) and choose the parameter set which achieves the best average accuracy or AUC.
Finally you will want to test the trained classifier on another dataset - evaluation set - which has not been part of the dataset used in crossvalidation.
Great answers but the answer also depends on the model usage. A small change in the training data row slice produces a large change in a validation set performance lowers my confidence that this is a good model. When using cross validation, I look at the variance of the performance to gauge if there was a lucky split or general instability. I do not ever blindly take the largest average of the cross validation performance.
When I encounter a model with a significant variance measured how you described, I may need to declare a problem if the model will be used in certain business contexts.
I may not be able to help the business understand the impact of the model, the best way to use the model in certain situations, the expected performance of the model over the long-term (some models are used once to score and some are scored millions of times over months or longer) and how to monitor the performance over the lifecycle.
In these cases, I may need to start from scratch - get more data (rows or columns), feature engineering, redefine my target (broader, narrower or different to achieve a similar business result), look for a different algorithm, or change how we approach this business problem. Or accept the risk, watch it closely, and be prepared to act accordingly.
Once again, dependent on the usage of the model if and how big a problem this may be. And how large a variance in model performance is also business context sensitive.
Most answers fail to address the following problem: even if you split your data into train and test, and perform k-fold cross validation on the training data to obtain the best model, your model's performance on the test data will depend on the initial "split" of training and test data. I can see only three solutions to this:
- Do not split the data into training and test and instead do k-fold cross validation on the full data set.
- Select the test data in some non-random way, for example, the last $K$ observations in the data set, if the data had collection timestamps, the logic being that you would like to maximise your performance on your latest observations.
- Use full data set with Cross Validation, but do a random shuffle split of the data as part of the cross validation