2
$\begingroup$

First I tried creating the training/testing datasets using sklearn train_test_split function like the following,

x_train, x_test, y_train, y_test = train_test_split(x_scaled, y, test_size=0.5, random_state = 1)

And on the second test I tried splitting the two datasets by half without any kind of randomization...

x_train, x_test, y_train, y_test = x_scaled[:int(total_rows/2)],x_scaled[int(total_rows/2):],y[:int(total_rows/2)],y[int(total_rows/2):]

On first test,

The model accuracy was like the following,

loss: 0.1951 - accuracy: 0.7057 - val_loss: 0.2101 - val_accuracy: 0.6540

and classification report was like this,

              precision    recall  f1-score   support

         0.0       0.55      0.76      0.64       864
         1.0       0.78      0.56      0.65      1263

    accuracy                           0.65      2127
   macro avg       0.66      0.66      0.65      2127
weighted avg       0.68      0.65      0.65      2127

On the second test, when I used splitted datasets, the model accuracy was pretty good, but test accuracy was below average...

loss: 0.1558 - accuracy: 0.7875 - val_loss: 0.5014 - val_accuracy: 0.5026

Classification report,

              precision    recall  f1-score   support

         0.0       0.47      0.80      0.59       965
         1.0       0.60      0.26      0.36      1162

    accuracy                           0.50      2127
   macro avg       0.54      0.53      0.48      2127
weighted avg       0.54      0.50      0.47      2127

I understand the second model is overfitting, that's why I'm getting poor test results... But in the real world the structure is gonna be kind of same... Like I'll have to use the model on fresh data while training it on older data... ( The rows are sorted or indexed by datetime in the datasets )

I'm kinda new to machine learning. So lil bit confused on this... Does the second test mean the model is not gonna perform that well like the first test in real world? Or what I'm doing wrong?

$\endgroup$

1 Answer 1

3
$\begingroup$

Normally overffited models will generalize poorly, because their parameters were estimated to follow the patterns found on your train set only. But why ?

The parameters/weights are estimated using gradient, if you don't know what it is 3Blue1Brown has a great video about that, think of it as a compass that points to the direction where your loss function converges to 0.

That direction is improved with the different patterns that your model finds in the data.

Although, you didn't shuffle your data, so some repeated patterns can show up in a sequence (e.g the first 100 images of the training set are cats) and the gradient will follow only those patterns until it finds a completely different pattern and realize: "Wait I'm in the wrong direction!" - So now it needs to recalculate the weights to follow the new pattern and that can happen nearly the end of the training loop, meaning that your model will not have time to learn the new pattern. Or it can't even find new patterns because those were only in the data in validation set when you truncate your inputs (x_train, x_test).

So your model will be very good to classify your training set data but only that data.

When you shuffle you show patterns in a random way so the model can update their weights (learning those patterns) in time and slowly converge to a minimum on your loss function.

There are other cases where model can overfit, a small dataset is one of those cases.

If it was confuse to understand, tell me I'll try to explain in a better way...

$\endgroup$
4
  • $\begingroup$ Thank you so much for explanation. I understand. But this model only works well when I train it on randomized training sets and test it on randomized test set. When I use it on randomized sets of first n rows and then test it on the rest of the rows. It underperforms again… $\endgroup$
    – Bucky
    Jul 11, 2022 at 10:54
  • $\begingroup$ So does retraining the model frequently after I receive a specific number of new rows of data can be the way to solve this issue? $\endgroup$
    – Bucky
    Jul 11, 2022 at 10:57
  • $\begingroup$ Btw I’m trying to build this model to filter out some noise on a short term algorithmic trading system I’m working on… Hope you get a better idea of the kind of data I’m talking about.. $\endgroup$
    – Bucky
    Jul 11, 2022 at 11:00
  • $\begingroup$ Did you tried to increase your N number of rows ? Like I said a small amount of samples can cause overfitting. I'm asking that because on the code you showed up you've a model with randomization and other without. I'm not 100% sure what is "short term algorithmic trading system" but Time Series are used on that ? $\endgroup$ Jul 13, 2022 at 21:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.