I have a project where I'm required to compute (using some regression) how long a task will take. From the definition of the business problem it is clear that there is some temporal relationship in the data so I need to split into train/test by some cutoff date (rather than randomly sampling 70/30 split from the data).

The issue I'm running into is that no matter what date I split by, the distribution of my target variable is different in the testing & training set. Earlier targets in the set are a lot larger, and as a result the model fit on the training set tends to way over-predict on the test set.

Any advice on how to tackle something like this?


It sounds likes you have non-stationary data which can be a challenge to model.

One option is to completely discard all older data and only train/test on newer data. This way the model will only capture the newer relationships. This approach assumes there is enough newer data to train a model.

Another option is heavily regularize the model. The goal of regularization is to increase generalization.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.