1
$\begingroup$

I've run random forest on my dataset (imbalanced, binary target class) and used cross validation to tune the parameter and use recursive feature elimination with cross-validation to get the subset of features.

Then I can present the optimal parameter setting as well as the set of features for model fitting.

But how about the probabilistic threshold?

How can I conclude a threshold as the important part of the final model?

Also using CV?

If this threshold is not stable among folds of CV, what to do next?

$\endgroup$
1
$\begingroup$

The probability threshold can be changed and you would obtain different results for accuracy / recall / precision / F1 score (the objective). You could see this as an aspect of the parameters you have tuned to obtain the best objective.

Similarly for CV, the out-of-training fold objective would be reported together with the other parameters, such as number of trees and predictor subset size. One reason I could think of regarding the instability of threshold across CV folds is that the CV folds were not split up using stratified sampling, resulting in different proportions of positives observed across the folds.

$\endgroup$
0
$\begingroup$

As a follow up to jkyh, I also would like to note that if the variance of your model is high that might also contribute to the changing metrics bewteen cv folds. You should examine bias/variance tradeoff.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.