4
$\begingroup$

I read for example in this answer: Does the performance of GBM methods profit from feature scaling?

that scaling doesn´t affect the performance of any tree-based method, not for lightgbm,xgboost,catboost or even decision tree.

When i do feature scaling and compare the rmse of a xgboost model without and with minmax scaling, i got a better rmse value with feature scaling. Here is the code:

from sklearn.preprocessing import MinMaxScaler
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error as MSE
import math
from math import sqrt
import pandas as pd
import numpy as np
from xgboost import XGBRegressor
import xgboost as xgb

data = pd.read_excel(r'C:...path.xlsx')
X = data.drop(['colA'], axis=1)
y = data['colA']


scaler = MinMaxScaler()
scaler.fit(X)
minmax_scaled_X = scaler.transform(X)
minmax_scaled_X
y = np.array(y).reshape(-1, 1)
scaler.fit(y)
minmax_scaled_y = scaler.transform(y)

from sklearn.model_selection import train_test_split
xtrain, xtest, ytrain, ytest = train_test_split(minmax_scaled_X, minmax_scaled_y, test_size =0.3, random_state=0, shuffle=True)

xg_reg = xgb.XGBRegressor(objective ='reg:squarederror', colsample_bytree = 0.7, learning_rate = 0.05,
                max_depth = 8, min_child_weight = 4, n_estimators = 600, subsample = 0.7)

xg_reg.fit(xtrain,ytrain)
preds = xg_reg.predict(xtest)
rmse = sqrt(MSE(ytest, preds))
print(rmse)

the result with min max scaling is 0.003, while the rmse without is about 3.8. I did the same with simple decision tree and got always a better result with minmax scaling.

Where is my mistake? In other posts like the link above, answers are about that it is not good to scale when using trees. Can I say, that min max scaling does have a positive effect on the rmse on my data?

$\endgroup$
4
$\begingroup$

You're also scaling $y$, then of course you are getting lower error. That question was regarding scaling $X$.

The same model will have very different error metrics when units on $y$ are changed: if I multiply all $y$ values by 100, the error will be 100 times larger, if I divide all $y$ values by 100 the error will be divided by 100.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Thank you very much. When and why do you scale x and y? or only x? Is there a rule or could you please recommend me some literature to read? $\endgroup$ – martin Jul 9 at 11:50
  • 1
    $\begingroup$ The standard thing to do is scaling X, as you can see in sklearn docs scikit-learn.org/stable/modules/preprocessing.html. I haven't ever seen a reason to scale y, but some to transform it $\endgroup$ – David Masip Jul 9 at 12:06
  • $\begingroup$ Thank you I got it. $\endgroup$ – martin Jul 12 at 20:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.