Python XGBoost predict_proba returns very high or low probabilities

Question

I trained my data with XGBoost in python with GridSearchCV as follows:

parameters = {'nthread':[6], 
              'objective':['binary:logistic'],
              'learning_rate': [0.01, 0.1],
              'max_depth': [5,8,13],
              'n_estimators': [200,500,1000,3000],
              'seed': [1337]}

xgb_model = xgb.XGBClassifier()

clf = GridSearchCV(xgb_model, parameters, n_jobs=-1, 
                   cv = StratifiedKFold(shuffle=True,n_splits=5), 
                   scoring='accuracy',
                   verbose=2, refit=True)

clf.fit(scaled_X_train.values, y_train)

On the test test I got 0.9 accuracy which is acceptable. However when I predict probabilities with predict_proba I saw that probabilities mostly lie between 0-0.1 and 0.9-1 ranges for 0 and 1 classes respectively.

Since I try to get scores based on the model, those dense probabilities are not so useful.

So what is the main reason of this dense probability distribution? Is this a bad thing? And how can I improve my workflow so that probabilities get wider score range?

Thanks in advance!

Answer 1

If you are after well calibrated scores (that is, the scores outputted by your model can be interpreted as probabilities in some sense, at least according to what you have observed empricially in your training set) then accuracy is not the right function to optimize (i would argue that it is never a good function to optimize, but I won't get into this). Accuracy can be optimized by providing scores that are not necessarily reflective of the empirical probabilities observed in your dataset: ex: suppose the true label = (1, 1, 0, 1) and you have two classifiers (0.51, 0.51, 0.3, 0.51) vs (0.9, 0.9, 0.1, 0.9). Both have the same accuracy assuming 0.5 threshold but clearly very different scores.

Switch your objective to log loss which is optimized only when you feed it well calibrated/true underlying probabilities. Furthermore, look into probability calibration (like platt/isotonic/multiple width binning) if you find that log loss performance is not satisfactory.

Answer 2

Yeah trees aren't the best way to get probabilities, they're very good at hard predictions though.

The decision tree probability estimates, which are a natural calculation from the frequencies at the leaves, can be systematically skewed towards 0 and 1, as the leaves are essentially dominated by one class

It's the way the algorithm works, the probabilities are biased toward the probable class per prediction.

Laplace estimate, m-estimate, and ensembles are able to overcome the bias in estimates arising from the axisparallel splits of decision trees, resulting in smoother estimates