6
$\begingroup$

I have a dependent variable, $Y$, that is made up of rates/percentages data, so each value is between $0$ and $1$. I was attracted to the xgboost library because it allows focusing in on specific subsets of the data in training itself, but I am stumped as how to perform regression on the data I have.

Normal OLS regression will produce outputs that will be over and under the [0,1] range if you do not change the likelihood to be the Beta distribution, or something else bounded in the same range, but will xgboost suffer the same mistake?

Any advice on trying to get this to work would be greatly appreciated by me.

$\endgroup$
5
$\begingroup$

In principle: yes, you will have the same problem as with OLS. However, since xgboost is tree-based (and by that non-parametric), you may get relatively accurate estimates, meaning that values which are below zero or above one would be rare (at least the problem should be less severe than with OLS). In this case you could simply restrict results to $\hat{y} \in [0,1]$. An alternative would be to do a multiclass classification task where you have 100 classes $y=[1\%,2\%,...,100\%]$. Boosting usually performs well on classification tasks.

Just a hint: xgboost can be somewhat "heavy" in data handling. Boosting tools such as lightgbm are a good alternative. I prefer lightgbm over xgboost.

$\endgroup$
  • 1
    $\begingroup$ +1 for suggesting lightgbm - for speed & memory usage. $\endgroup$ – bradS Aug 6 at 8:08
2
$\begingroup$

You could use the reg:logistic objective function. https://xgboost.readthedocs.io/en/latest/parameter.html#learning-task-parameters

$\endgroup$
  • $\begingroup$ But this command only takes classes as input and not floats, right? Or am I mistaken? $\endgroup$ – Peter Aug 6 at 15:17
  • 2
    $\begingroup$ No, the "reg" indicates it's trying to fit regression, just with the logistic link. It turns out that even binary:logistic will happily work with inputs between 0 and 1, and in fact is the exact same fitting objective. The difference between the two is just the default evaluation metric. datascience.stackexchange.com/questions/9802/… github.com/bmreiniger/datascience.stackexchange/blob/master/… $\endgroup$ – Ben Reiniger Aug 6 at 15:37
  • $\begingroup$ Ah, okay... this might be a good solution for the problem. $\endgroup$ – Peter Aug 6 at 15:59
0
$\begingroup$

I agree with all those answers, it is a regression case. You can convert it into classification case if you are able to split your target values into several classes.

But I would also try to transform target values this way:

old_target = function(new_target)
new_target = inverse_function(old_target)

For example, softmax looks promising. It will give you another range of target values:

(-inf, inf)

Then, you train XGBRegressor on this transformed target.

In general, the type of function() depends on the target values distribution.

$\endgroup$
  • $\begingroup$ So... what you're saying is a softmax transformation on my bounded [-1,1] values would spread it over the real number line (-inf, inf)? Wouldn't that cause problems with the values being spread out so wide that some of them will simply be too ungainly/too big/small for calculations? $\endgroup$ – Coolio2654 Aug 10 at 20:39
  • $\begingroup$ > what you're saying is a softmax transformation on my bounded [-1,1] values would spread it over the real number line (-inf, inf)? Inverse softmax transformation. $\endgroup$ – avchauzov Aug 11 at 4:01
  • $\begingroup$ > Wouldn't that cause problems with the values being spread out so wide that some of them will simply be too ungainly/too big/small for calculations? >> That's why I said "the type of function() depends on the target values distribution". Take 'wider softmax' in range (-2; 2). $\endgroup$ – avchauzov Aug 11 at 4:03
  • 1
    $\begingroup$ Also, check lightgbm: > binary, binary log loss classification (or logistic regression). Requires labels in {0, 1}; see cross-entropy application for general probability labels in [0, 1] multi-class classification application It has the option you need: lightgbm.readthedocs.io/en/latest/Parameters.html In this case you may need to convert your [-1, 1] to [0, 1] and then convert to classification case. $\endgroup$ – avchauzov Aug 11 at 4:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.