LinearRegression with multiple binary features sometimes performs poorly

Question

I have a dataset comprising a number of binary features which are the dummies (as in, pd.get_dummies()) of categorical features. SalePrice is my target variable.

I'm literally just fitting a sklearn LinearRegression model with that data a thousand times to get an average of the score, and I'm getting a weird result. The relevant bit of my code looks like this:

import numpy as np
scores = np.array([])

for i in range(1000):

    x3_train, x3_test, y3_train, y3_test = train_test_split(
            df3.drop('SalePrice', axis=1),
            df3.SalePrice,
            test_size=0.33
    )

    lr3 = LinearRegression()
    lr3.fit(x3_train, y3_train)

    scores = np.insert(scores, 0, lr3.score(x3_test, y3_test))

print(scores.mean())

Now the weird result is that the average result is super poor, because every so often the model just tanks completely but most of the time performs "reasonably" (still terrible but that's not a surprise as it's incredibly basic and not tuned at all, I'm just comparing the effect of treating a set of features in different ways). For example the first 30 runs generated these scores:

0     5.907010e-01
1     6.044523e-01
2     5.178049e-01
3     5.622240e-01
4     5.810432e-01
5     5.131722e-01
6     5.772946e-01
7     4.674152e-01
8     4.962015e-01
9     4.887872e-01
10    5.144772e-01
11    5.676829e-01
12    5.122566e-01
13    5.453985e-01
14    5.355022e-01
15    5.888459e-01
16    5.552912e-01
17    5.615658e-01
18    5.472429e-01
19    5.810185e-01
20    5.334900e-01
21    5.493619e-01
22    5.567195e-01
23    5.514374e-01
24    4.916478e-01
25    4.580718e-01
26    5.286095e-01
27    5.761865e-01
28    5.638573e-01
29   -1.809208e+24
Name: lr3, dtype: float64

I guess my question is what is likely to be happening on that 30th run through such that the model performs so poorly? I'm comparing this model to others that treat the data differently (e.g. simply encode using .astype('category').cat.codes) and whilst there's relatively minor variations in the "usual" range of scores (they're all sort of 0.44 - 0.63) those other models don't have this occasional complete tanking.

Answer 1

You should always consider normalizing your output to some predefined range, otherwise there is a possibility of the gradients exploding as the loss will be of high magnitudes. It also becomes hard to output such a wide range. Try transforming your output using some StandardScaler, or a RobustScaler if there are significant outliers, and try again.

Answer 2

Prior to jumping to any conclusions, some questions that immediately comes to my mind:

• What is the correlation between your features and target?
• Do you have any numerical features too?
• How large is your feature space (how many independent variables)?
• How about cardinality of your categorical features (levels)?
• Intuitively, are your features are good indicators for predicting SalePrice?
• How your regressor performs, in terms of the distributions of residual?
• Last but not least, have you tried any other regressors?

Initial Guess: It would be that a simple Linear Regression won't work because there is no linear correlation between your features and target (see Assumptions of Linear Regressions).

Practical Suggestion: I would suggest trying a quick and dirty Gradient Boosting Trees for Regression (either sklearn or XGboost or Catboost implementation) and see if you notice any immediate improvement. From your explanation I see that you have quick a few categorical features that you encoded using One-Hot-Encoding (OHE) method via pd.get_dummies() in pandas. I have personally experienced that OHE is not a good idea for most of problems esp. when your have a lot of categorical features and they present high cardinality (i.e. many levels in each categorical feature), and if you search you find such examples that people struggle using OHE. Anyways, here are two very quick implementation of Catboost Regressor in Kaggle 1, 2 to have a quick start. Good thing about Catboost is that one does not need to encode categorical features, you can pass then as it is, you only need to give the column index of your categorical features (let me know if you have struggle make Catboost up and running!).