1
$\begingroup$

I'm working on a data set that has continuous dependent variable. I used XG Boost to model the dependent variable. However, when I transformed the dependent variable by applying Log transformation and then modeled it using XG Boost, the results were drastically improved. I am getting results close to 100% of actual on Test Data. Is there any explanation to this ?

$\endgroup$
1
  • $\begingroup$ "results close to 100%" means what metric? $\endgroup$
    – Ben Reiniger
    Oct 25, 2021 at 20:19

1 Answer 1

1
$\begingroup$

It's actually good practice to perform some type of scaling of your continuous variables. The box-cox transformation (or positive variable) or the yeo-johnson transformation (which adapts the box-cox transformation) are in the class of power transorms implemented in scikit-learn.

I you look at the implementation, you see that the log transformation that you applied is a particular case of box-cox.

Gradient boosting includes an optimisation step that will rely on computing gradients of your loss function. A recurrent theme in ML is how gradient descent have a harder time finding optimal solution when the data has a high order of magnitude. Essentially the information is there at high order of magnitude but the optimisation space is "stretched" - scaling your data brings everything closer if you will (that's a really gross explanation of what happens).

So essentially you've done a great job at making the most out of your data by transforming it to get a better model.

And this is actually an excellent case because, you often hear that when using decision tree you should not scale the data, and by that what is meant are the features. When using a decision tree model for regression you should absolutely consider scaling the target.

$\endgroup$
1
  • $\begingroup$ I normally think of "scaling" to mean a linear transform, and applying those to the target is also of no use (unless there are some computational issues). But a nonlinear transform like the logarithm will affect the value at each leaf (the mean of the logarithms is not the log of the mean), and so could potentially improve performance. That said, I would've expected enough boosting iterations to wash that difference out for the most part... $\endgroup$
    – Ben Reiniger
    Oct 25, 2021 at 20:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.