3
$\begingroup$

I understand that the loss metric can be used as linear, or log, or other things. This is documented at http://lightgbm.readthedocs.io/en/latest/Parameters.html?highlight=logloss#metric-parameters

I would like to understand how LightGBM works on variables with different scale. In other words, is it necessary for me to harmonize scale when running LightGBM? (I am used to linear regression where you need to get into linear scale.)

If I had inputs x1, x2, x3, output y and some noise N then here are a few examples of different scales.

  • $y = x1 + x2 + x3 + N $
  • $y = exp(x1 + x2 + x3 + N) $
  • $y = log(x1 + x2 + x3 + N) $
  • $y = sqrt(x1 + x2 + x3 + N) $
  • $y = log(x1 * x2 * x3 * N) $
$\endgroup$
5
$\begingroup$

Generally, in tree-based models the scale of the features does not matter. This is because at each tree level, the score of a possible split will be equal whether the respective feature has been scaled or not.

You can think of it like here: We're dealing with a binary classification problem and the feature we're splitting takes values from 0 to 1000. If you split it on 300, the samples <300 belong 90% to one category while those >300 belong 30% to one category. Now imaging this feature is scaled between 0 and 1. Again, if you split on 0.3, the sample <0.3 belong 90% to one category while those >0.3 belong 30% to one category.

So you've changed the splitting point but the actual distribution of the samples remains the same regarding the target variable.

$\endgroup$
  • $\begingroup$ Your example discusses multiplying a value by 1000, which is something that linear regression already does seamlessly. Are you saying that f(x), where f is a strictly increasing function of x, also has the same prediction value of x? $\endgroup$ – William Entriken Aug 8 '17 at 14:31
  • $\begingroup$ Yes, if you scale your features using any strictly monotonic function, you should get the same predictions. There may be a few slight differences. These occur due to the splitting points chosen and where the new samples lie in that space. Anyway, the differences should be negligible. $\endgroup$ – Stergios Aug 8 '17 at 15:07
  • $\begingroup$ Thank you. And how does this deal with scaling on the /output/ side? e.g. y=log(x1+x2+x3+N) $\endgroup$ – William Entriken Aug 11 '17 at 21:21
  • $\begingroup$ Scaling the output variable does affect the learned model, and actually it is a nice idea to try if you want to ensemble many different LightGBM (or any regression) models. From my practical experience, the predictions based on a scaled output variable and on the original one will be highly correlated between each other (i.e. >0.98-0.99). $\endgroup$ – Stergios Aug 14 '17 at 5:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.