0
$\begingroup$

I am making some ML methods (RF, RNN, MLP) to predict a time series value 'y' based on features 'X' and not the time series 'y' itself. My question is regarding the bias I might be including since I am doing a simple random train-test-split for the fit and evaluation process, so I am using data from different days (past and future) and not spliting by time. Is it valid for this prediction process, or even that I am not using the time series to predict future values, I am still introducing bias since I am using features which also are time series themselves since I go them daily. I tried both ways and got far better results using the simple random train-test-split, so I got suspicious.

$\endgroup$

1 Answer 1

0
$\begingroup$

The answer depends on whether there is autocorrelation in the y target variable which is not accounted for in your X regressors. In other words: if you know all the X values and you are trying to predict the current time step y value, would it help you at all to know the previous time steps' y values? If so, there is autocorrelation in y that is not reducible to your X features, and your simple random train-test-split is not advisable.

A simple example: suppose X is just temperature, and y is depth of snow on the ground each morning in, say, some particular spot in northern Canada. Even though X is a pretty good predictor of y, the relevant autocorrelation still holds, in the sense that even if I know X, my prediction of today's y value will be much, much better if I also know yesterday's y value.

Of course, in real world situations, we usually never really know for sure whether there is the kind of autocorrelation described above. If you suspect it might be there, then play it safe and use a train/test or cross validation method that respects the time domain by training only on data prior to the test split(s).

$\endgroup$
2
  • $\begingroup$ I believe my data clearly has autocorrelation, in that case using random train-test-split would add a bias that might help me get better results on the test set but would lack of real power for future values right ? I made models using both methods of splits and random ones made the metrics extremely good while time base split dropped significantly. I guess thats an insight also for guessing time base split is the correct way to go $\endgroup$
    – JmML
    Jan 28, 2022 at 16:56
  • $\begingroup$ Yes, that would explain why you get better results on the simple-random test set than on future values. $\endgroup$
    – Ceph
    Jan 30, 2022 at 20:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.