7
$\begingroup$

I have a dependent variable, $Y$, that is made up of rates/percentages data, so each value is between $0$ and $1$. I was attracted to the xgboost library because it allows focusing in on specific subsets of the data in training itself, but I am stumped as how to perform regression on the data I have.

Normal OLS regression will produce outputs that will be over and under the [0,1] range if you do not change the likelihood to be the Beta distribution, or something else bounded in the same range, but will xgboost suffer the same mistake?

Any advice on trying to get this to work would be greatly appreciated by me.

Improve this question
$\endgroup$

3 Answers 3

Reset to default
7
$\begingroup$

In principle: yes, you will have the same problem as with OLS. However, since xgboost is tree-based (and by that non-parametric), you may get relatively accurate estimates, meaning that values which are below zero or above one would be rare (at least the problem should be less severe than with OLS). In this case you could simply restrict results to $\hat{y} \in [0,1]$. An alternative would be to do a multiclass classification task where you have 100 classes $y=[1\%,2\%,...,100\%]$. Boosting usually performs well on classification tasks.

Just a hint: xgboost can be somewhat "heavy" in data handling. Boosting tools such as lightgbm are a good alternative. I prefer lightgbm over xgboost.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ +1 for suggesting lightgbm - for speed & memory usage. $\endgroup$
    – bradS
    Aug 6, 2019 at 8:08
5
$\begingroup$

You could use the reg:logistic objective function. https://xgboost.readthedocs.io/en/latest/parameter.html#learning-task-parameters

Edit: You need to use either the internal API for this, or use the sklearn API's XGBRegressor with objective='reg:logistic' (or binary:logistic).
Do not use XGBClassifier, as it will label-encode the target!

Improve this answer
$\endgroup$
3
  • $\begingroup$ But this command only takes classes as input and not floats, right? Or am I mistaken? $\endgroup$
    – Peter
    Aug 6, 2019 at 15:17
  • 2
    $\begingroup$ No, the "reg" indicates it's trying to fit regression, just with the logistic link. It turns out that even binary:logistic will happily work with inputs between 0 and 1, and in fact is the exact same fitting objective. The difference between the two is just the default evaluation metric. datascience.stackexchange.com/questions/9802/… github.com/bmreiniger/datascience.stackexchange/blob/master/… $\endgroup$
    – Ben Reiniger ♦
    Aug 6, 2019 at 15:37
  • $\begingroup$ Ah, okay... this might be a good solution for the problem. $\endgroup$
    – Peter
    Aug 6, 2019 at 15:59
0
$\begingroup$

I agree with all those answers, it is a regression case. You can convert it into classification case if you are able to split your target values into several classes.

But I would also try to transform target values this way:

old_target = function(new_target)
new_target = inverse_function(old_target)

For example, softmax looks promising. It will give you another range of target values:

(-inf, inf)

Then, you train XGBRegressor on this transformed target.

In general, the type of function() depends on the target values distribution.

Improve this answer
$\endgroup$
4
  • $\begingroup$ So... what you're saying is a softmax transformation on my bounded [-1,1] values would spread it over the real number line (-inf, inf)? Wouldn't that cause problems with the values being spread out so wide that some of them will simply be too ungainly/too big/small for calculations? $\endgroup$
    – Coolio2654
    Aug 10, 2019 at 20:39
  • $\begingroup$ > what you're saying is a softmax transformation on my bounded [-1,1] values would spread it over the real number line (-inf, inf)? Inverse softmax transformation. $\endgroup$
    – andrewchauzov
    Aug 11, 2019 at 4:01
  • $\begingroup$ > Wouldn't that cause problems with the values being spread out so wide that some of them will simply be too ungainly/too big/small for calculations? >> That's why I said "the type of function() depends on the target values distribution". Take 'wider softmax' in range (-2; 2). $\endgroup$
    – andrewchauzov
    Aug 11, 2019 at 4:03
  • 1
    $\begingroup$ Also, check lightgbm: > binary, binary log loss classification (or logistic regression). Requires labels in {0, 1}; see cross-entropy application for general probability labels in [0, 1] multi-class classification application It has the option you need: lightgbm.readthedocs.io/en/latest/Parameters.html In this case you may need to convert your [-1, 1] to [0, 1] and then convert to classification case. $\endgroup$
    – andrewchauzov
    Aug 11, 2019 at 4:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.