0
$\begingroup$

I am training a Decision tree regressor on the famous Boston House Price dataset. I read that tree based models are fairly immune to scaling so I tried to see practically. Before scaling I was getting MAE = 2.61. After scaling using StandardScaler, I got worse results which is quite surprising as scaling the data should result in an improvement. Also if not an improvement, I was hoping to see the same result as I got without scaling since I am using a tree based model. Also there are no categorical variables in this dataset and all the variables included in the scale_list a large difference in their min and max values hence have been included. I have not included and binary (1's and 0's)variables.

 train_x, test_x, train_y, test_y = train_test_split(data2, y, test_size = 0.2, random_state = 
 69)

#STANDARDIZING THE DATA
scale_list = ['CRIM', 'NOX', 'RM', 'AGE', 'DIS', 'TAX', 'PTRATIO', 'B', 'LSTAT']

train_x_stand = train_x.copy()
test_x_stand = test_x.copy()

for i in scale_list:

   scale = StandardScaler().fit(train_x_stand[[i]])
   train_x_stand[i] = scale.transform(train_x_stand[[i]])
   test_x_stand[i] = scale.transform(test_x_stand[[i]])

Any help would be appreciated!

$\endgroup$
1
$\begingroup$

As you well mentioned, tree-based models are not sensitive to feature scaling, but on the contrary it might help with the convergency of finding the minimum in the optimization on boosted models

I replicated your code and I found pretty much the same metrics in both scaling and no scaling versions of the model.

from sklearn.datasets import load_boston
from sklearn.tree import DecisionTreeRegressor
from sklearn.pipeline import Pipeline
from sklearn.compose import make_column_transformer
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import mean_absolute_error


boston= load_boston()

X, y = pd.DataFrame(boston.data, columns=boston.feature_names), pd.DataFrame(boston.target)

train_x, test_x, train_y, test_y = train_test_split(X, y, test_size = 0.2, random_state = 69)

scale_list = ['CRIM', 'NOX', 'RM', 'AGE', 'DIS', 'TAX', 'PTRATIO', 'B', 'LSTAT']


preprocessor = make_column_transformer((StandardScaler(),scale_list), remainder="passthrough")


smodel = Pipeline([("scaler",preprocessor),
                   ("model", DecisionTreeRegressor(random_state=42))]).fit(train_x, train_y)

tmodel = Pipeline([("model", DecisionTreeRegressor(random_state=42))]).fit(train_x, train_y)

print(f"metric with scale features: {mean_absolute_error(test_y, smodel.predict(test_x))}\nmetric with no scale features:{mean_absolute_error(test_y, tmodel.predict(test_x))}")

metric with scale features: 2.6941176470588237
metric with no scale features:2.662745098039216

I dare to think that something is wrong with the way you are applying the scaling. I recommend you to check it or use pipelines.

Also I was unsure if you used Ensemble or a simple DT, but in both cases remember to set the random_state in order to have reproducible results.

Hope it helps!

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.