0
$\begingroup$

I'm working on a data set that has continuous dependent variable. I used XG Boost to model the dependent variable. However, when I transformed the dependent variable by applying Log transformation and then modeled it using XG Boost, the results were drastically improved. I am getting results close to 100% of actual on Test Data. Is there any explanation to this ?

$\endgroup$
0
$\begingroup$

It's actually good practice to perform some type of scaling of your continuous variables. The box-cox transformation (or positive variable) or the yeo-johnson transformation (which adapts the box-cox transformation) are in the class of power transorms implemented in scikit-learn.

I you look at the implementation, you see that the log transformation that you applied is a particular case of box-cox.

Gradient boosting includes an optimisation step that will rely on computing gradients of your loss function. A recurrent theme in ML is how gradient descent have a harder time finding optimal solution when the data has a high order of magnitude. Essentially the information is there at high order of magnitude but the optimisation space is "stretched" - scaling your data brings everything closer if you will (that's a really gross explanation of what happens).

So essentially you've done a great job at making the most out of your data by transforming it to get a better model.

And this is actually an excellent case because, you often hear that when using decision tree you should not scale the data, and by that what is meant are the features. When using a decision tree model for regression you should absolutely consider scaling the target.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.