I have a fundamental question about train/test split for time series. Let me give a simple example to illustrate my question, which is actually related to a more complex problem.


Suppose I have the following dataset and I want to learn to predict how fast a worker can perform a task, as given in the column task_duration_minutes:

date worker_id task_id task_duration_minutes
2010-01-01 1 1 75
2010-01-01 2 2 75
2010-01-01 3 2 60
2010-01-02 1 1 50
2010-01-03 3 2 45
etc... etc... etc... etc...

(I also have a table with features related to the worker_id, and a table with features related to the task_id which I can join on this table, but the worker features and task features are not relevant for the problem.)

Note that new workers can have their first task late in this time series, and the duration between tasks is not equal. Hence, it is presumably not the date column that is important but rather the number of times a certain worker has previously done a certain task. Let's feature-engineer the column worker_task_experience to extract that information:

date worker_id task_id task_duration_minutes worker_task_experience
2010-01-01 1 1 75 0
2010-01-01 2 2 75 0
2010-01-01 3 2 60 0
2010-01-02 1 1 50 1
2010-01-03 3 2 45 1
etc... etc... etc... etc... etc...

Question: How to properly split this dataset for cross validation

I see two options:

  1. One could view this dataset as a time series for each worker_id and the common practice for time series is to train/test-split as TimeSeriesSplit.
  2. However, the target task_duration_minutes is not considered to be dependent on the date, but rather on the worker_task_experience, and hence I can disregard the date column and view this problem as a regular regression problem, for which I can do a regular cross validation using e.g. KFold.

As long as I am certain that my features do not contain information about the future, I can't see 2) is not acceptable. What do you think?

I am looking forward to your input.

  • $\begingroup$ I want to make sure that I understand the question correctly. Given user_id, and task_id, you want to estimate the duration of the task, next time it is completed, is this right? If so, let's say you have a user_id and item_id with 100 tasks completed, would you need an ML model to predict 101st task completion or duration of 100th task is close enough? Also, I suspect the biggest error and perhaps the biggest value gain is in predicting durations for early tasks, 1st, 2nd, 3rd... but I don't see how it is possible to do that with the approach you suggested. $\endgroup$
    – Akavall
    Apr 10, 2022 at 22:46
  • $\begingroup$ My true problem is more complicated and contain business problems that I cannot explain here, so forgive me if this example is not completely thought through. I hope my questions about time Series splitning is still clear. But to answer your question: I see your concern, since task_id now completely defines the task. However, in reality, I never see the exact same task beeing done but only similar tasks. So I never see 100 tasks completed by the same worker. The 101th task may have different external factors. $\endgroup$
    – Jasoba
    Apr 11, 2022 at 5:17


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.