Low validation accuracy when not using shuffled datasets

Question

First I tried creating the training/testing datasets using sklearn train_test_split function like the following,

x_train, x_test, y_train, y_test = train_test_split(x_scaled, y, test_size=0.5, random_state = 1)

And on the second test I tried splitting the two datasets by half without any kind of randomization...

x_train, x_test, y_train, y_test = x_scaled[:int(total_rows/2)],x_scaled[int(total_rows/2):],y[:int(total_rows/2)],y[int(total_rows/2):]

On first test,

The model accuracy was like the following,

loss: 0.1951 - accuracy: 0.7057 - val_loss: 0.2101 - val_accuracy: 0.6540

and classification report was like this,

              precision    recall  f1-score   support

         0.0       0.55      0.76      0.64       864
         1.0       0.78      0.56      0.65      1263

    accuracy                           0.65      2127
   macro avg       0.66      0.66      0.65      2127
weighted avg       0.68      0.65      0.65      2127

On the second test, when I used splitted datasets, the model accuracy was pretty good, but test accuracy was below average...

loss: 0.1558 - accuracy: 0.7875 - val_loss: 0.5014 - val_accuracy: 0.5026

Classification report,

              precision    recall  f1-score   support

         0.0       0.47      0.80      0.59       965
         1.0       0.60      0.26      0.36      1162

    accuracy                           0.50      2127
   macro avg       0.54      0.53      0.48      2127
weighted avg       0.54      0.50      0.47      2127

I understand the second model is overfitting, that's why I'm getting poor test results... But in the real world the structure is gonna be kind of same... Like I'll have to use the model on fresh data while training it on older data... ( The rows are sorted or indexed by datetime in the datasets )

I'm kinda new to machine learning. So lil bit confused on this... Does the second test mean the model is not gonna perform that well like the first test in real world? Or what I'm doing wrong?

Answer 1

Normally overffited models will generalize poorly, because their parameters were estimated to follow the patterns found on your train set only. But why ?

The parameters/weights are estimated using gradient, if you don't know what it is 3Blue1Brown has a great video about that, think of it as a compass that points to the direction where your loss function converges to 0.

That direction is improved with the different patterns that your model finds in the data.

Although, you didn't shuffle your data, so some repeated patterns can show up in a sequence (e.g the first 100 images of the training set are cats) and the gradient will follow only those patterns until it finds a completely different pattern and realize: "Wait I'm in the wrong direction!" - So now it needs to recalculate the weights to follow the new pattern and that can happen nearly the end of the training loop, meaning that your model will not have time to learn the new pattern. Or it can't even find new patterns because those were only in the data in validation set when you truncate your inputs (x_train, x_test).

So your model will be very good to classify your training set data but only that data.

When you shuffle you show patterns in a random way so the model can update their weights (learning those patterns) in time and slowly converge to a minimum on your loss function.

There are other cases where model can overfit, a small dataset is one of those cases.

If it was confuse to understand, tell me I'll try to explain in a better way...