model predicting probability close to 50 for positive classes in imbalanced training dataset

Question

I have a binary classification model where I am predicting the positive class which is only 10% of whole training data set. The issue with this imbalanced data set is my model is predicting probability not more than 0.6 for the positive class. Why is it happening?

I know it is due to imbalanced data set and there are some answers which suggests how to deal with imbalanced data sets.

I have dealt with imbalanced data sets before and have never encountered such a case where probability for a class is not more than 0.6.

Answer 1

More than the probability, what you can consider is the confusion matrix you get after applying the threshold on the probability. If, for example, your threshold is 0.5 and your model is perfectly classifying your validation examples, you shouldn't have a problem.

If you take average log loss as your training metric, with equal weights for positive and negative classes, your average probability values will get shifted towards the dominant side in the training set.

Summarizing, probability values in themselves may not be a cause of worry. Focus more on imbalanced data specific metrics like precision, recall(sensitivity) specificity, F1 score or Area Under the ROC curve.

Answer 2

your model is not able to discriminate between the positive and negative classes that good, not just because of the data distribution, but also the predictors in both of your classes are not that much efficient to seperate the same.

I would suggest to do some feature engineering on top of that, may be removing some can boost the probability.

If you are able to share the dataset, then I can also take a look at it.

Share your results anyway.