High error machine learning regressor algorithm in Python - XGBOOST Regressor

Question

I have a dataframe with real state data from florida, it includes single apartments and buildings data:

'TRUE_SITE_CITY': The city where the building is. variable: Miami, Aventura...;
'CONDO_FLAG': If it is a condominium or not, variable: yes/no;
'BEDROOM_COUNT': Number of total bethrooms, variable: integuer,
'BUILDING_actual_AREA': The area of the entire building, or apartment in the case that there are only one apartment or house. variable: integuer;
'FLOOR_COUNT': Number of the floors that the building has;
'DOR_CODE_CUR': the type of the building. Variable: categorical;
'UNIT_COUNT': Number of apartments or houses in the building. Variable: integuer;
'YEAR_BUILT': Year that the building or house or apartment was build: Variable: categorical;
'public_transport_min_distance': I have calculated the nearest stations of the public transport;
'Price': The variable that I want to predict.Variable: integer.

I have done an exploratory data analysis and I have dropped some data that has null values and some data that was incorrect. Also I have dropped the values with outliers.

The basic statistics of the price column (targeted column):

I have checked the categorical features and they have enough variables in each one to keep they in the model.

I have done a pipeline to make a one hot encoder for the categorical values and a standard standardisation for the numerical values. In it I have include a XGBOOST regression:

from xgboost import XGBRegressor
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import cross_validate
from sklearn import metrics
from sklearn import preprocessing, feature_extraction
from sklearn.pipeline import Pipeline
from sklearn import preprocessing, feature_extraction
from sklearn.pipeline import make_pipeline, make_union
from mlxtend.feature_selection import ColumnSelector
from sklearn.preprocessing import StandardScaler
from category_encoders import OneHotEncoder

x_numeric = df_x[['BEDROOM_COUNT','BATHROOM_COUNT',
       'HALF_BATHROOM_COUNT', 'FLOOR_COUNT','UNIT_COUNT','public_transport_min_distance','BUILDING_actual_AREA']]

x_categorical = df_x[['TRUE_SITE_CITY','CONDO_FLAG','YEAR_BUILT']]

categorical_col = x_categorical.columns

numeric_col = x_numeric.columns

estimator_pipeline = Pipeline([
    ('procesador', procesing_pipeline),
    ('estimador', estimator)
])

score2 = cross_validate(estimator_pipeline, X= df_x, y= df_y, scoring=scoring,return_train_score=False, cv=5,n_jobs=2)

But I am obtaining a high error. The mean value of the price is almost 200.000 and the error that I obtain is:

I have done feature selection using RFE but I obtain a high error as well.

Also I have run it doing RandomizedSearchCV

from sklearn.model_selection import RandomizedSearchCV

params = {"estimator__learning_rate"    : [0.05, 0.10, 0.15, 0.20, 0.25, 0.30 ] ,
 "estimator__max_depth"        : [ 3, 4, 5, 6, 8, 10, 12, 15],
 "estimator__min_child_weight" : [ 1, 3, 5, 7 ],
 "estimator__gamma"            : [ 0.0, 0.1, 0.2 , 0.3, 0.4 ],
 "estimator__colsample_bytree" : [ 0.3, 0.4, 0.5 , 0.7 ] }

random_search = RandomizedSearchCV(
    estimator=estimator_pipeline, 
    param_distributions=params, cv=5, refit=True,
    scoring="neg_mean_squared_error", n_jobs= 3,
    return_train_score=True,
    n_iter=50)

But I obtain a similar error value.

What could I do?

Answer 1

• It is not necessary to one-hot-encode your categorical features when using tree-based methods. Basically the idea is that the tree has to make many splits to figure out the category. Instead, you can use ordinal encoder (even if categories are not ordered).

• I also would not have set 'YEAR_BUILT' as a categorical variable, even though it is discrete. By one-hot encoding this variable, you end up with a very splitted dataset where the tree has to consider every YEAR_BUILT as a subset and it probably has too few observations to do so. By considering it as continuous it enables the algorithm to determine 2 or 3 pivot years that are relevant (changed currency, economic crisis, ...).

I think the second point can really improve performance.