I fit my dataset to the random forest classifier and found that the model performance would vary among different sets of train and test data split. As what I have observed, it would jump from 0.67 to 0.75 in AUC under ROC curve (fitted by the same model under same setting of parameters) and the underlying range may be wider than that. So what is the issue behind this phenomena and how to deal with this problem? As my understanding, cross validation is used for a specific split of train and test data.
$\begingroup$
$\endgroup$
7
5 Answers
$\begingroup$
$\endgroup$
2
While training, your model will not have the same output when you train with different parts of the dataset. Cross validation is used to help negate this, by rotating the training and validation sets and training more.
Your dataset most likely has high variance, given the large jump in accuracy based on different validation sets. This means that the data is spread out, and can result in overfitting the model. You can imagine an overfitted model like this:
The green line represents the overfitted model.
Common techniques to reduce overfitting in random forests is k-fold Cross Validation, with k being between 5 and 10, and growing a larger forest.
-
$\begingroup$ Thanks @ChristianSafka. I currently split my dataset into train and test sets. Then fit the model to the train set and using the estimator to fit the test set for prediction. Could you please advice how to conduct the cross validation as what you mentioned? $\endgroup$– LUSAQXCommented Jan 7, 2017 at 9:04
-
$\begingroup$ Specific implementation of cross validation will vary from library to library. For example, if you are using sklearn and Python, here is an example: scikit-learn.org/stable/modules/… $\endgroup$ Commented Jan 9, 2017 at 9:25
$\begingroup$
$\endgroup$
2
What you are experiencing is not a problem, but rather an inherent attribute of all classifiers. The performance of a classifier depends on the training set, therefore the performance will vary with different training sets.
To find the best parameters for a specific classifier you will therefore want to vary training and test split (such as in crossvalidation) and choose the parameter set which achieves the best average accuracy or AUC.
Finally you will want to test the trained classifier on another dataset - evaluation set - which has not been part of the dataset used in crossvalidation.
-
$\begingroup$ So you mean to use cross validation to (1) find the best parameter set and (2) the dataset was separated into 3 part: train set, validation set and test set (the first two were used for cross validation)? So I just found the best classifier based on cross validation (train set plus validation set) and then fit the test set. But how can I ensure this classifier outperformed on the test set than any other inferior classifiers? $\endgroup$– LUSAQXCommented Jan 7, 2017 at 9:21
-
1$\begingroup$ First, you split the dataset into development (70%) and evaluation(30%) set. Then you use the development set repeatedly to build your model. In each repetition, you choose a different test-train split (non-overlapping). Then you choose the best models (including parameters) and evaluate it using the evaluation set. For more information on crossvalidation, please read users.isr.ist.utl.pt/~wurmd/Livros/school/… page 22 ff $\endgroup$ Commented Jan 7, 2017 at 17:22
$\begingroup$
$\endgroup$
Most answers fail to address the following problem: even if you split your data into train and test, and perform k-fold cross validation on the training data to obtain the best model, your model's performance on the test data will depend on the initial "split" of training and test data. I can see only three solutions to this:
- Do not split the data into training and test and instead do k-fold cross validation on the full data set.
- Select the test data in some non-random way, for example, the last $K$ observations in the data set, if the data had collection timestamps, the logic being that you would like to maximise your performance on your latest observations.
- Use full data set with Cross Validation, but do a random shuffle split of the data as part of the cross validation
$\begingroup$
$\endgroup$
Great answers but the answer also depends on the model usage. A small change in the training data row slice produces a large change in a validation set performance lowers my confidence that this is a good model. When using cross validation, I look at the variance of the performance to gauge if there was a lucky split or general instability. I do not ever blindly take the largest average of the cross validation performance.
When I encounter a model with a significant variance measured how you described, I may need to declare a problem if the model will be used in certain business contexts.
I may not be able to help the business understand the impact of the model, the best way to use the model in certain situations, the expected performance of the model over the long-term (some models are used once to score and some are scored millions of times over months or longer) and how to monitor the performance over the lifecycle.
In these cases, I may need to start from scratch - get more data (rows or columns), feature engineering, redefine my target (broader, narrower or different to achieve a similar business result), look for a different algorithm, or change how we approach this business problem. Or accept the risk, watch it closely, and be prepared to act accordingly.
Once again, dependent on the usage of the model if and how big a problem this may be. And how large a variance in model performance is also business context sensitive.
$\begingroup$
$\endgroup$
I don't think there is no definite answer for this issue. But my impression is that you need to know some robustness of your model (selection) in the sense that the (selected) model performances are decent and stable regardless of train/test splits.
RandomForestClassifier(n_estimators=1000,criterion='gini',max_features='auto' ,oob_score=True,class_weight = 'balanced')$\endgroup$max_depthhelp to get stable model performance given a specific train and test set, i.e. AUC will lie in a narrow range after several iteration of the model fitting under a fixed set of parameters. $\endgroup$