# High error machine learning regressor algorithm in Python - XGBOOST Regressor

I have a dataframe with real state data from florida, it includes single apartments and buildings data:

'TRUE_SITE_CITY': The city where the building is. variable: Miami, Aventura...;
'CONDO_FLAG': If it is a condominium or not, variable: yes/no;
'BEDROOM_COUNT': Number of total bethrooms, variable: integuer,
'BUILDING_actual_AREA': The area of the entire building, or apartment in the case that there are only one apartment or house. variable: integuer;
'FLOOR_COUNT': Number of the floors that the building has;
'DOR_CODE_CUR': the type of the building. Variable: categorical;
'UNIT_COUNT': Number of apartments or houses in the building. Variable: integuer;
'YEAR_BUILT': Year that the building or house or apartment was build: Variable: categorical;
'public_transport_min_distance': I have calculated the nearest stations of the public transport;
'Price': The variable that I want to predict.Variable: integer.


I have done an exploratory data analysis and I have dropped some data that has null values and some data that was incorrect. Also I have dropped the values with outliers.

The basic statistics of the price column (targeted column):

I have checked the categorical features and they have enough variables in each one to keep they in the model.

I have done a pipeline to make a one hot encoder for the categorical values and a standard standardisation for the numerical values. In it I have include a XGBOOST regression:

from xgboost import XGBRegressor
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import cross_validate
from sklearn import metrics
from sklearn import preprocessing, feature_extraction
from sklearn.pipeline import Pipeline
from sklearn import preprocessing, feature_extraction
from sklearn.pipeline import make_pipeline, make_union
from mlxtend.feature_selection import ColumnSelector
from sklearn.preprocessing import StandardScaler
from category_encoders import OneHotEncoder

x_numeric = df_x[['BEDROOM_COUNT','BATHROOM_COUNT',
'HALF_BATHROOM_COUNT', 'FLOOR_COUNT','UNIT_COUNT','public_transport_min_distance','BUILDING_actual_AREA']]

x_categorical = df_x[['TRUE_SITE_CITY','CONDO_FLAG','YEAR_BUILT']]

categorical_col = x_categorical.columns

numeric_col = x_numeric.columns

estimator_pipeline = Pipeline([
])

score2 = cross_validate(estimator_pipeline, X= df_x, y= df_y, scoring=scoring,return_train_score=False, cv=5,n_jobs=2)


But I am obtaining a high error. The mean value of the price is almost 200.000 and the error that I obtain is:

I have done feature selection using RFE but I obtain a high error as well.

Also I have run it doing RandomizedSearchCV

from sklearn.model_selection import RandomizedSearchCV

params = {"estimator__learning_rate"    : [0.05, 0.10, 0.15, 0.20, 0.25, 0.30 ] ,
"estimator__max_depth"        : [ 3, 4, 5, 6, 8, 10, 12, 15],
"estimator__min_child_weight" : [ 1, 3, 5, 7 ],
"estimator__gamma"            : [ 0.0, 0.1, 0.2 , 0.3, 0.4 ],
"estimator__colsample_bytree" : [ 0.3, 0.4, 0.5 , 0.7 ] }

random_search = RandomizedSearchCV(
estimator=estimator_pipeline,
param_distributions=params, cv=5, refit=True,
scoring="neg_mean_squared_error", n_jobs= 3,
return_train_score=True,
n_iter=50)


But I obtain a similar error value.

What could I do?

• It is not necessary to one-hot-encode your categorical features when using tree-based methods. Basically the idea is that the tree has to make many splits to figure out the category. Instead, you can use ordinal encoder (even if categories are not ordered).

• I also would not have set 'YEAR_BUILT' as a categorical variable, even though it is discrete. By one-hot encoding this variable, you end up with a very splitted dataset where the tree has to consider every YEAR_BUILT as a subset and it probably has too few observations to do so. By considering it as continuous it enables the algorithm to determine 2 or 3 pivot years that are relevant (changed currency, economic crisis, ...).

I think the second point can really improve performance.

• Thank you for your answer! I am going to test your ideas. One question: The 'estimated_current_price' column I have created it getting the selling price multiplied by the US inflation in relation between the selling year and 2020. Do you think this is a good estimation? – J.C Guzman Apr 12 at 12:38
• This question could almost deserve its own post. I will split my comment into two parts, first the case of the corrected price using inflation and then the classic 'feature' year variable. – Rusoiba Apr 12 at 13:47
• - If you create a feature with inflation based on 2020 you basically transform your dataset "as every building was built in 2020" and that will merge decisional paths. For example, you have two very similar apartments (2 rooms, 100ft squared) but the second was built 30 years before. Your algorithm will consider that the features have the same 'importance' for both (because now, the year is not distinctive criteria anymore). So if you think that apartment pricing is pretty constant in term of what is the most important etc, this is a good assumption (also removing complexity !) – Rusoiba Apr 12 at 13:52
• - If you use year as numerical variable, the algorithm is able to create a path like "if Year > 2010, having WiFi is highly valuable", and "if Year < 2010, Wifi is not important but having a TV is". So basically, you enable the tree to split the equation on year, defining itself what are the important features for each subset in time (1950-1970, 1970-1990, ...). I hope my answer is clear, otherwise don't hesitate to create a new question and I will make a proper answer. – Rusoiba Apr 12 at 13:56