Why after adding categorical data the Linear Regression fails?

Question

Based on a training set we applied a simple Linear Regression on some attributes that all were numeric.

Now we have more attributes in terms of categories and of course we applied one-hot-encoding to transform the categories to binary attributes

Take for example this simple python code:

X_train, X_test, y_train, y_test = train_test_split(X, y, train_size = 0.8, test_size=0.2)

model = LinearRegression(normalize=True).fit(X_train, y_train)

printErrorMetrics(trueTargets=y_test, predictions=model.predict(X_test))

When the table X has only the original numeric attributes the scores from the printErrorMetrics function (RMSE, etc.) are all good enough

We were expecting better results after adding the one-hot-encoded categories but the results are so worse that the method does not seem to work.

Are we missing anything?

Do we need to preprocess the data after adding the one-hot-encoded columns/attributes?

Answer 1

One possible reason is that when you use one-hot-encoding for categorical data, you should set the intercept property in the function to be False:

model = LinearRegression(fit_intercept=False, normalize=True).fit(X_train, y_train)

This will avoid the dummy variable trap: http://www.algosome.com/articles/dummy-variable-trap-regression.html

You could also use dummy encoding to avoid this problem: http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.LabelEncoder.html