0
$\begingroup$

I am in doubt when to use strict time-series cross validation and when to use kfold. I have the following situation, which, I believe, is an edge-case between time series and normal data:

I have a small dataset which is a couple of thousand rows. The data is collected over time, but I only have a few observations for each shop (specified by shop_id) which are note evenly spaced. For the majority of shops, I only have a single observation and therefore, treating each shop as a separate time series is not meaningful. I have feature-engineered the feature called last_sales which give the last sales for that shop_id. Suppose the first 5 rows look like this:

time shop_id #fetures# last_sales sales
1 1 nan 8
1 2 nan 3
3 1 8 4
5 3 nan 2
9 2 3 2

where #features# are a number of other features.

I want to predict the sales in the future for a known or unknown shop_id.

My question: When validating my model, should I use time-series splitting or is it ok to use kfold ? Note, in the end I am not interested in knowing my models performance over time. I am only interested in estimating the model performance in the future.

My thoughts:

If I should be very correct, I would think that I should use time-series splitting to take into account that some correlations between a feature and the target may change over time. On the other hand, it seems silly that when testing the performance at time = 4 at shop_id = 1 my model is not allowed to be trained on e.g. the data point time = 8 at shop_id = 2. How bad would it be if I just treat these rows as observations not recorded over time and use normal KFold cross validation utilizing my entire dataset. I emphasize, I want to estimate my model performance for future predictions. Not the model performance in the past, where I had fewer data points available.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

I believe a time-series does not make sense here if you don't have repeated measurements for all shops. You even wrote most shops have only one observation which would make time-series methods impossible.

What you should do here is proper feature engineering that aggregates the temporal information to one unique shop_id. Thus, assume the last_sales is measured at different points in time for different shops. You could then include the time difference between today and the time of measurement as a feature. Or even simpler, include the time of last_sales as a dummy variable (eg. time=8 - yes/no).

Another feature could be the difference in sales between the last_sales measurement and the previous sales figures. You can really get creative in the feature engineering process. With the resulting dataset you could then do KFold-cross-validation.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thanks for your answer. This was also how I thought it. So it is ok to have e.g. t=20 in in my training set and t=2 in my test set? (Both for the same or different shops) $\endgroup$
    – Jasoba
    Commented Mar 2, 2023 at 22:08
  • $\begingroup$ Well for the same shop this should not happen because my idea was to have each shop_id only once in your dataset. But your concern here is justified. Kfold might lead to data leakage were information from the future is leaking into the training. Thus in your example it is a problem when the information in t=20 for a shop contains information from another shop from t=2. If you fear that the data from different shops is interdependent you might be safer to restrict your test set to all shops in the most recent time points $\endgroup$
    – danielOh
    Commented Mar 3, 2023 at 23:25
  • $\begingroup$ Thanks. I’m wondering if my fears can be tested in any way, or if it is just up to me to decide on my assumptions. I suppose there must be some research on this? $\endgroup$
    – Jasoba
    Commented Mar 6, 2023 at 16:03
  • 1
    $\begingroup$ Your performance on the test set should be better if you have data leakage. Which is not good in this case because it is just a result from a suboptimal data split and not from well-learned model paramters. You could exclude the most recent observations from your dataset as a hold-out test set. If you do cross-validation with the remaining data your performance on the validation set should be much better than on your hold-out test set. $\endgroup$
    – danielOh
    Commented Mar 7, 2023 at 8:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.