I have an imbalanced data set where positives are just 10% of the whole sample. I am using logistic regression and random forest for classification. While comparing the results of these models, I have found that the probability output of logistic regression ranges between [0,1] while that of random forest ranges between [0, 0.6]. I cannot share the data set but my doubt is around the working of these algorithms. How can random forest generate probability less than 0.6?
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
3
To have a probability of 1 in a RF, it means that your algorithm can construct a leaf containing only positive sample. Since it doesn't, this means that your features are not explaining the variance of the output or that your algorithm is under-fitted.
I suggest that you try optimize the hyper-parameters of your RF by using cross-validation and use some oversampling to reduce the bias in your dataset.
-
$\begingroup$ Makes sense. Why logistic regression doesn't suffer from imbalanced data set? Is it also possible that due to imbalanced data set logistic probabilities are also biased? $\endgroup$ Jul 23, 2020 at 9:18
-
$\begingroup$ I assume you are talking about the output on your train set. While the random forest prediction depend on the output value (mean of outputs in a leaf) , the logistic regression only depends on the coefficients. Of course the coefficients are learnt based on the output but this affects more the distribution of the output than its range. But again consider optimizing your hyperparameters, you should at least by overfitting, be able to catch a bigger range. $\endgroup$– mirimoJul 23, 2020 at 11:01
-
1$\begingroup$ Okay. I have tried different hyper-parameters to overfit but I am getting the same range. I guess in my case it is related to imbalanced data set. Thus, oversampling might help. $\endgroup$ Jul 23, 2020 at 12:30