4
$\begingroup$

I know that feature scaling is an important pre-processing step for creating artificial neural network models.

But what about Gradient Boosting Machines, such as LightGBM, XGBoost or CatBoost? Does their performance profit from feature scaling? If so, why and how?

Improve this question
$\endgroup$
2

1 Answer 1

Reset to default
5
$\begingroup$

Scaling doesn't affect the performance of any tree-based method, not for lightgbm, xgboost, catboost or even a decision tree.

This post that elaborates on the topic, but mainly the issue is that decision trees split the feature space based on binary decisions like "is this feature bigger than this value?", and if you scale your data, the decisions might look different, as they are done on the scaled space, but the results should be the same.

As an example, a decision tree should split in the same way the data if you change units (a particular case of scaling). Let's say you want to use weight of a person to predict if someone is under 18 (as a binary classifier). If given the weight in grams, the decision tree might do something like: if weight < 5000 grams, then the person is under 18. If you change the units to kilograms, the decision tree will do: if weight < 5 kg, then the person is under 18.

To sum up, the splits will be equivalent under any scaling or, in general, any increasing linear transformation.

Improve this answer
$\endgroup$
3
  • 2
    $\begingroup$ Monotone transformations are a little trickier, since the splitting point may be chosen at or between two values of a feature in the training set, depending on the implementation. The model will perform the same on the transformed training set, but an unseen test set may get different predictions from the original model and the transformed-training model. $\endgroup$
    – Ben Reiniger ♦
    Commented Jul 7, 2020 at 14:43
  • $\begingroup$ You're right. It shouldn't happen with linear transformations though, right? $\endgroup$
    – David Masip
    Commented Jul 7, 2020 at 16:09
  • 2
    $\begingroup$ For increasing (i.e. positive slope) linear functions like most/all scalers, yes everything about the model will be the same in the transformed space. See also datascience.stackexchange.com/q/32638/55122 $\endgroup$
    – Ben Reiniger ♦
    Commented Jul 7, 2020 at 17:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.