Splitting Temporal Data

Question

I have a project where I'm required to compute (using some regression) how long a task will take. From the definition of the business problem it is clear that there is some temporal relationship in the data so I need to split into train/test by some cutoff date (rather than randomly sampling 70/30 split from the data).

The issue I'm running into is that no matter what date I split by, the distribution of my target variable is different in the testing & training set. Earlier targets in the set are a lot larger, and as a result the model fit on the training set tends to way over-predict on the test set.

Any advice on how to tackle something like this?

Answer 1

It sounds likes you have non-stationary data which can be a challenge to model.

One option is to completely discard all older data and only train/test on newer data. This way the model will only capture the newer relationships. This approach assumes there is enough newer data to train a model.

Another option is heavily regularize the model. The goal of regularization is to increase generalization.