I have a dataset comprising a number of binary features which are the dummies (as in,
pd.get_dummies()) of categorical features. SalePrice is my target variable.
I'm literally just fitting a sklearn LinearRegression model with that data a thousand times to get an average of the score, and I'm getting a weird result. The relevant bit of my code looks like this:
import numpy as np scores = np.array() for i in range(1000): x3_train, x3_test, y3_train, y3_test = train_test_split( df3.drop('SalePrice', axis=1), df3.SalePrice, test_size=0.33 ) lr3 = LinearRegression() lr3.fit(x3_train, y3_train) scores = np.insert(scores, 0, lr3.score(x3_test, y3_test)) print(scores.mean())
Now the weird result is that the average result is super poor, because every so often the model just tanks completely but most of the time performs "reasonably" (still terrible but that's not a surprise as it's incredibly basic and not tuned at all, I'm just comparing the effect of treating a set of features in different ways). For example the first 30 runs generated these scores:
0 5.907010e-01 1 6.044523e-01 2 5.178049e-01 3 5.622240e-01 4 5.810432e-01 5 5.131722e-01 6 5.772946e-01 7 4.674152e-01 8 4.962015e-01 9 4.887872e-01 10 5.144772e-01 11 5.676829e-01 12 5.122566e-01 13 5.453985e-01 14 5.355022e-01 15 5.888459e-01 16 5.552912e-01 17 5.615658e-01 18 5.472429e-01 19 5.810185e-01 20 5.334900e-01 21 5.493619e-01 22 5.567195e-01 23 5.514374e-01 24 4.916478e-01 25 4.580718e-01 26 5.286095e-01 27 5.761865e-01 28 5.638573e-01 29 -1.809208e+24 Name: lr3, dtype: float64
I guess my question is what is likely to be happening on that 30th run through such that the model performs so poorly? I'm comparing this model to others that treat the data differently (e.g. simply encode using
.astype('category').cat.codes) and whilst there's relatively minor variations in the "usual" range of scores (they're all sort of 0.44 - 0.63) those other models don't have this occasional complete tanking.