I use random forest to train on my data (My data had imbalance in the target class, i.e. rare 1 and abundant 0). I face 3 issues about the stability of estimator and its prediction power. I think these problems could be common on many machine learning algorithms.
- I found the
ROC_AUC_Scorehighly changeable when I resample the training set (the rest was test set). It can vary from $0.85$ to $0.45$ while changing the training set.
- The parameter tuning can also cause the move of the estimator and
ROC_AUC_Scorebut the effect was weaker than the first case above.
- When running some iteration of model fitting, the results were also different from each other but the effect was the weakest.
For 2, 3, I think we can get the best parameter setting and fitting results by recording each iteration and parameter tuning. (Maybe can do it more efficiently, please advice)
Please also advice how to deal with the first problem to make the fitting convinced and reliable? Cross Validated?