25
$\begingroup$

The below predict function is giving -ve values as well so it cannot be probabilities. 

param <- list(max.depth = 5, eta = 0.01,  objective="binary:logistic",subsample=0.9)
bst <- xgboost(param, data = x_mat, label = y_mat,nround = 3000)

pred_s <- predict(bst, x_mat_s2)

I google & tried pred_s <- predict(bst, x_mat_s2,type="response") but it didn't work.

Question

How to predict probabilities instead?

Improve this question
$\endgroup$
4
  • $\begingroup$ Doesn't it output probabilities by default with the settings you used? I mean: have you examined pred_s and you are certain those are not probabilities? $\endgroup$
    – kpb
    Sep 8, 2015 at 11:58
  • $\begingroup$ No its having negative values. Probability should vary between 0 to 1. $\endgroup$
    – GeorgeOfTheRF
    Sep 8, 2015 at 12:05
  • $\begingroup$ I don't see any obvious issues. (Although, I'm more familiar with the python wrapper). Have you tried adding outputmargin=F to the predict function? If somehow the outputmargin is set to T, it will return the value before the logistic transformation. $\endgroup$
    – inversion
    Sep 10, 2015 at 17:24
  • 1
    $\begingroup$ For Python, you can copy predict_proba implementation from sklearn API: github.com/dmlc/xgboost/blob/master/python-package/xgboost/… $\endgroup$
    – Anton Tarasenko
    Jan 19, 2018 at 15:39

4 Answers 4

Reset to default
22
$\begingroup$

Just use predict_proba instead of predict. You can leave the objective as binary:logistic.

Improve this answer
$\endgroup$
7
  • 4
    $\begingroup$ If this were Python and not R, then this answer might be sensible. Wrong language. $\endgroup$
    – B_Miner
    Aug 15, 2016 at 18:30
  • 2
    $\begingroup$ oops! thanks @B_Miner. I'm not deleting this answer as it might be helpful for others that will make the same mistake and think we're talking about python.. $\endgroup$
    – ihadanny
    Aug 16, 2016 at 8:49
  • $\begingroup$ For me this does not do the trick datascience.stackexchange.com/questions/14527/… $\endgroup$
    – Georg Heiler
    Nov 8, 2016 at 7:59
  • 6
    $\begingroup$ xgboost does not have a predict_proba function $\endgroup$
    – Ashoka Lella
    Aug 24, 2017 at 19:26
  • 2
    $\begingroup$ XGBoost Classifier does has a predict_proba option xgboost.readthedocs.io/en/latest/python/python_api.html $\endgroup$
    – Paul Bendevis
    Dec 24, 2019 at 19:07
17
$\begingroup$

Know I'm a bit late, but to get probabilities from xgboost you should specify multi:softmax objective like this:

xgboost(param, data = x_mat, label = y_mat,nround = 3000, objective='multi:softprob')

From the ?xgb.train:

multi:softprob same as softmax, but output a vector of ndata * nclass, which can be further reshaped to ndata, nclass matrix. The result contains predicted probabilities of each data point belonging to each class.

Improve this answer
$\endgroup$
3
  • 2
    $\begingroup$ Thanks. How is this loss function different from binary:logistic for binary classification? $\endgroup$
    – GeorgeOfTheRF
    Nov 12, 2015 at 7:30
  • 3
    $\begingroup$ It's just a generalization of logistic function for multi-class case, should be no significant difference. $\endgroup$
    – cyberj0g
    Nov 12, 2015 at 7:39
  • $\begingroup$ I used it and it worked well, but I had an issue. I'm facing a warning "Parameters: { scale_pos_weight } might not be used.", which comes form the fact I use multi:softprob (source). Why my unbalanced data, it seems that this param is really important (if I go for a classic binary classification), and currently, when I change it, it has no impact because it's not taken into account. $\endgroup$
    – Adept
    Aug 13, 2020 at 8:06
3
$\begingroup$

After the prediction:

pred_s <- predict(bst, x_mat_s2)

You can get the probability by:

pred_s$data

If this is a binary classification then pred_s$data includes prob.0, prob.1, response.

So you can get prob.1 by:

pred_s$data$prob.1
Improve this answer
$\endgroup$
0
$\begingroup$

Many years late, but noticed this; and this is how I do it:

pred_s <- predict(bst, x_mat_s2, type="prob")

Typically then I then work with probability of my upper class "1":

pred_s[,2]
Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.