2
$\begingroup$

I use random forest to train on my data (My data had imbalance in the target class, i.e. rare 1 and abundant 0). I face 3 issues about the stability of estimator and its prediction power. I think these problems could be common on many machine learning algorithms.

  1. I found the ROC_AUC_Score highly changeable when I resample the training set (the rest was test set). It can vary from $0.85$ to $0.45$ while changing the training set.
  2. The parameter tuning can also cause the move of the estimator and ROC_AUC_Score but the effect was weaker than the first case above.
  3. When running some iteration of model fitting, the results were also different from each other but the effect was the weakest.

For 2, 3, I think we can get the best parameter setting and fitting results by recording each iteration and parameter tuning. (Maybe can do it more efficiently, please advice)

Please also advice how to deal with the first problem to make the fitting convinced and reliable? Cross Validated?

Many thanks

Improve this question
$\endgroup$
3
  • 2
    $\begingroup$ Can you please put your code here ? $\endgroup$
    – enterML
    Commented Nov 24, 2016 at 2:30
  • $\begingroup$ Reducing the depth of trees may be the solution to your problem $\endgroup$
    – RpyGamer
    Commented Nov 24, 2016 at 13:11
  • 1
    $\begingroup$ Can you describe your imbalance and data set a bit more? (sample sizes, # predictors, etc) $\endgroup$
    – TBSRounder
    Commented Nov 25, 2016 at 15:09

3 Answers 3

Reset to default
1
$\begingroup$

You should definitely use nested cross-validation for model selection and performance estimation. I have also found that the AUC of the precision-recall (PR) curve (compared to that of the ROC curve) to be a better, that is, more stable, estimator of the performance of my Random Forest classifiers when I have a highly unbalanced dataset; there is research on this topic, showing that the AUC of the PR curve is more informative than that of the ROC curve. You can use average_precision_score() in scikit-learn to use the PR AUC score. Along the lines of resampling the data, you could try approaches like EasyEnsenble and BalanceCascade; search for the papers titled "Exploratory Undersampling for Class-Imbalance Learning" and "Learning from Imbalanced Data" for more information.

Improve this answer
$\endgroup$
0
$\begingroup$

I think, it's a classic class imbalance problem. If it's possible, collect more data of the class with sparse amount of data or if you have sufficient amount of data, omit some of the data of the class with large amount of data.

Improve this answer
$\endgroup$
2
  • $\begingroup$ I tried stratified sampling or other sampling methods with sample weights. But when it comes to the dependency of estimators on the training set, does cross validation work? $\endgroup$
    – LUSAQX
    Commented Nov 23, 2016 at 23:57
  • $\begingroup$ Cross Validation help us use all of our data that too without overfitting the training data on our model. Therefore, in my thinking, cross-validation should help get a better dependency of estimators on the training set. $\endgroup$
    – Vivek Khetan
    Commented Nov 24, 2016 at 0:02
0
$\begingroup$

I am not sure what the ratio of imbalance is, and with which package of RF or the one you wrote by yourself you have experienced , but I think you can try it again in two ways to solve the problem of stability:

  1. It's obviously that your problem of imbalance is severe , you should keep it balanced as much as you can do.
  2. If you have not sampled features for different trees in RF, namely different features for different trees , you can try it.

Hopes this will help you, good luck !

Improve this answer
$\endgroup$
2
  • $\begingroup$ This is my Python code for random forest:rf = RandomForestClassifier(n_estimators=3000,oob_score=True,class_weight = 'balanced') $\endgroup$
    – LUSAQX
    Commented Nov 24, 2016 at 2:39
  • $\begingroup$ I have not used python package , and not sure feature space varies from tree to tree. You should check and figure it out . By the way what is the ratio of your imbalance ? $\endgroup$
    – joe
    Commented Nov 24, 2016 at 2:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.