4
$\begingroup$

I am training a logistic regression model on a dataset with only numerical features. I performed the following steps:-

1.) heatmap to remove collinearity between variables

2.) scaling using StandarScaler

3.) cross validation after splitting, for my baseline model

4.) fitting and predicting

Below is my code:-

# SPLITTING 
train_x, test_x, train_y, test_y = train_test_split(data2, y, test_size = 0.2, random_state = 
69)

#MODEL INSTANCE
model = LogisticRegression(random_state = 69)

# SCALING
train_x2 = train_x.copy(deep = True)
test_x2 = test_x.copy(deep = True)

s_scaler = StandardScaler()
s_scaler.fit(train_x2)
s_scaled_train = s_scaler.transform(train_x2)
s_scaled_test = s_scaler.transform(test_x2)

# BASELINE MODEL
cross_val_model2 = -1 * cross_val_score(model, s_scaled_train, train_y, cv = 5,
                              n_jobs = -1, scoring = 'neg_mean_squared_error')
s_score = cross_val_model2.mean()

# FITTING AND PREDICTING
model.fit(s_scaled_train, train_y)
pred = model.predict(s_scaled_test)
mse = mean_squared_error(test_y, pred)

CV score is 0.06 and score after fitting and predicting is 0.23. I find this weird as CV is a measure of how good your model performs. So I should atleast get a score equal to the CV score right?

$\endgroup$
2
  • $\begingroup$ 0.23 score is better than 0.06, rt? Am I missing some thing here? Sorry for my ignorance. $\endgroup$
    – Venkat
    Aug 6, 2021 at 6:05
  • $\begingroup$ My bad for not mentioning the metric. I am using mean_squared_error. So 0.23 is worse than 0.06 $\endgroup$
    – spectre
    Aug 6, 2021 at 8:17

3 Answers 3

3
$\begingroup$

A couple additional points:

  • With MSE or any error measure, the value is relative to the data. This implies that one cannot know what the difference between 0.06 and 0.23 represents: this difference might be very large or very small, it all depends on the distribution of the target.
  • As Sammy mentions, the small size of the data is likely to cause high variations depending on the split. In order to understand the difference it would be useful to observe the performance variation across CV folds, for example by calculating the standard deviation: if the variation is high then the model is not stable, and that could explain the high difference.
$\endgroup$
11
  • $\begingroup$ Regarding your first point, this is a binary classification problem so target has only 1's and 0's. Regarding your second point, the 5 score's of cv are 0.148, 0.333, 0.222, 0.230, 0.153. I don't think there's a lot of variance. $\endgroup$
    – spectre
    Aug 6, 2021 at 16:10
  • $\begingroup$ Also if the dataset is small (like my case), should I even perform CV? $\endgroup$
    – spectre
    Aug 6, 2021 at 16:11
  • $\begingroup$ @spectre if your task is binary classification then you should probably not use MSE, a classification measure like F1-score makes more sense. I think the CV values have a quite high variance actually, and 0.23 is totally consistent with these values. Also their mean is much higher than 0.06, are these values obtained from a different run of the CV process? $\endgroup$
    – Erwan
    Aug 6, 2021 at 16:39
  • $\begingroup$ About whether or not performing CV: it never hurts to do CV, if only to observe how stable the performance is. It's not strictly necessary either, but in any case this question is not related to the size of the dataset: the problem with a small dataset (or a model too complex for the size) is the high risk of overfitting, this is what happens here and causes the instability. $\endgroup$
    – Erwan
    Aug 6, 2021 at 16:45
  • 1
    $\begingroup$ @spectre not really :) However it's correct that sometimes a model can look good if one doesn't carefully evaluate, or if one wants to cheat and present a biased version of the results. Model instability is something that should be checked for instance. But in your case I suspect that part of the problem is using MSE instead of a classification measure: every error is counted as 1 and you have a small dataset, so a small difference of a few more/less errors is going to change the score a lot. $\endgroup$
    – Erwan
    Aug 6, 2021 at 18:30
5
$\begingroup$

With only 210 samples the difference might be caused by your train and test data not being from the same underlying distribution. That is, using holdout-CV to estimate model performance on such a small dataset can be susceptible to how you split your data. You can test this by re-running the pipeline with a different train/test-split and see if results differ.

Instead, for small datasets I suggest to apply nested k-fold CV. You can read more about in, for example, Bias in Error Estimation When Using Cross-Validation for Model Selection. Here you can find an implementation in scikit learn.

$\endgroup$
6
  • $\begingroup$ I set test_size = 0.5 and my cv score is 0.18 and my final score is 0.19. This might be called an acceptable score but isn't it weird that I am using 50% of my data for train and test? I mean 50% of data for train is too less and 50% of data for test is too much. Also changing the test_size to get an "acceptable answer" does not seem like a good idea as I can possibly get a good score for any kind of data and model by changing this parameter, even if my data and/or model is not good. $\endgroup$
    – spectre
    Aug 6, 2021 at 9:21
  • $\begingroup$ @spectre What I meant is to split your data with the same test_size but different random_state and see if results change. However, the fact that your k-fold CV and holdout CV scores become more similar with test_size=0.5 supports my hypothesis too. Therefore, I recommend applying nested (k-fold) CV instead of holdout CV. That is, do not use any holdout train/test-split. $\endgroup$
    – Jonathan
    Aug 6, 2021 at 9:35
  • 1
    $\begingroup$ @spectre I've added a link to an implementation in scikit learn. Have a look. $\endgroup$
    – Jonathan
    Aug 6, 2021 at 9:54
  • $\begingroup$ I tried random_state = 100 instead of 69 and now I got proper results. My CV mse score is 0.149 and final mse score is 0.029. This is weird. This means that I can make any model good (even if my data is bad or the model preprocessing steps or hyperparameters are bad) just by changing random_state. Also the 5 cv values are 0.333, 0.074, 0.185, 0.000, 0.153 $\endgroup$
    – spectre
    Aug 6, 2021 at 18:22
  • 1
    $\begingroup$ @spectre it does not make your model good. It only demonstrates that your validation procedure is improper with regards to your dataset. $\endgroup$
    – Jonathan
    Aug 7, 2021 at 7:12
2
$\begingroup$

No expert here, so dont take my words too close to heart, but I can think of 2 reasons for this. First, the CV, if split into too many sections, can not work prperly (depending on the data size basically), even cause overfitting. Second, The metric you are using: mean_squared_error will not necessarily show you the results that most of us want (I dont know if thats the case with you). Try using R2 score, might give you different results. Good luck! hope this helps even a tiny bit

$\endgroup$
4
  • $\begingroup$ My dataset size is (210, 7). Also I am using cv = 5 which is not a large value to cause any kind of overfitting (AFAIK). Usually it is advised to use some kinda loss function as a scoring method when performing CV. That is the reason I chose mse. $\endgroup$
    – spectre
    Aug 6, 2021 at 8:49
  • 1
    $\begingroup$ 210 is a very small dataset. Any loss in data from such a small amount can result in some unusual performance, even if all else is as expected and correct. Did you trey to play around with hyperparameters of the model and of the CV, try to maximize the performance? In any case, if it is at all possible, get more data, you will most likely get a more accurate model $\endgroup$ Aug 6, 2021 at 8:52
  • $\begingroup$ I am using a logistic regression model so there's not much to play around as far as hyperparameters are considered. I'll try to set cv = 3 to see if I get any improvement. Other than that I don't know what else I can change $\endgroup$
    – spectre
    Aug 6, 2021 at 9:03
  • $\begingroup$ After choosing cv = 3, my cv score is 0.135 and my final score is 0.235. some improvement. $\endgroup$
    – spectre
    Aug 6, 2021 at 9:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.