I fit my dataset to the random forest classifier and found that the model performance would vary among different sets of train and test data split. As what I have observed, it would jump from 0.67 to 0.75 in AUC under ROC curve (fitted by the same model under same setting of parameters) and the underlying range may be wider than that. So what is the issue behind this phenomena and how to deal with this problem? As my understanding, cross validation is used for a specific split of train and test data.

  • 2
    $\begingroup$ what do you mean by same parameters but different training sets? maybe you can describe your process in a bit more detail to make it clear what is going on. $\endgroup$ – oW_ Jan 5 '17 at 21:13
  • $\begingroup$ Parameters of the random forest like n_estimator, criterion='Gini' etc are fixed but the input (train set) was varying from each split of the dataset. $\endgroup$ – LUSAQX Jan 5 '17 at 21:17
  • $\begingroup$ how did you set the parameters? $\endgroup$ – oW_ Jan 5 '17 at 21:25
  • $\begingroup$ RandomForestClassifier(n_estimators=1000,criterion='gini',max_features='auto' ,oob_score=True,class_weight = 'balanced') $\endgroup$ – LUSAQX Jan 5 '17 at 21:27
  • $\begingroup$ I can find max_depth help to get stable model performance given a specific train and test set, i.e. AUC will lie in a narrow range after several iteration of the model fitting under a fixed set of parameters. $\endgroup$ – LUSAQX Jan 5 '17 at 21:44

While training, your model will not have the same output when you train with different parts of the dataset. Cross validation is used to help negate this, by rotating the training and validation sets and training more.

Your dataset most likely has high variance, given the large jump in accuracy based on different validation sets. This means that the data is spread out, and can result in overfitting the model. You can imagine an overfitted model like this:

Green line is the overfitted model

The green line represents the overfitted model.

Common techniques to reduce overfitting in random forests is k-fold Cross Validation, with k being between 5 and 10, and growing a larger forest.

  • $\begingroup$ Thanks @ChristianSafka. I currently split my dataset into train and test sets. Then fit the model to the train set and using the estimator to fit the test set for prediction. Could you please advice how to conduct the cross validation as what you mentioned? $\endgroup$ – LUSAQX Jan 7 '17 at 9:04
  • $\begingroup$ Specific implementation of cross validation will vary from library to library. For example, if you are using sklearn and Python, here is an example: scikit-learn.org/stable/modules/… $\endgroup$ – Christian Safka Jan 9 '17 at 9:25

What you are experiencing is not a problem, but rather an inherent attribute of all classifiers. The performance of a classifier depends on the training set, therefore the performance will vary with different training sets.

To find the best parameters for a specific classifier you will therefore want to vary training and test split (such as in crossvalidation) and choose the parameter set which achieves the best average accuracy or AUC.

Finally you will want to test the trained classifier on another dataset - evaluation set - which has not been part of the dataset used in crossvalidation.

  • $\begingroup$ So you mean to use cross validation to (1) find the best parameter set and (2) the dataset was separated into 3 part: train set, validation set and test set (the first two were used for cross validation)? So I just found the best classifier based on cross validation (train set plus validation set) and then fit the test set. But how can I ensure this classifier outperformed on the test set than any other inferior classifiers? $\endgroup$ – LUSAQX Jan 7 '17 at 9:21
  • 1
    $\begingroup$ First, you split the dataset into development (70%) and evaluation(30%) set. Then you use the development set repeatedly to build your model. In each repetition, you choose a different test-train split (non-overlapping). Then you choose the best models (including parameters) and evaluate it using the evaluation set. For more information on crossvalidation, please read users.isr.ist.utl.pt/~wurmd/Livros/school/… page 22 ff $\endgroup$ – Nikolas Rieble Jan 7 '17 at 17:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.