I am in doubt when to use strict time-series cross validation and when to use kfold. I have the following situation, which, I believe, is an edge-case between time series and normal data:
I have a small dataset which is a couple of thousand rows. The data is collected over time, but I only have a few observations for each shop (specified by
shop_id) which are note evenly spaced. For the majority of shops, I only have a single observation and therefore, treating each shop as a separate time series is not meaningful. I have feature-engineered the feature called
last_sales which give the last sales for that
shop_id. Suppose the first 5 rows look like this:
#features# are a number of other features.
I want to predict the
sales in the future for a known or unknown
My question: When validating my model, should I use time-series splitting or is it ok to use kfold ? Note, in the end I am not interested in knowing my models performance over time. I am only interested in estimating the model performance in the future.
If I should be very correct, I would think that I should use time-series splitting to take into account that some correlations between a feature and the target may change over time.
On the other hand, it seems silly that when testing the performance at
time = 4 at
shop_id = 1 my model is not allowed to be trained on e.g. the data point
time = 8 at
shop_id = 2. How bad would it be if I just treat these rows as observations not recorded over time and use normal KFold cross validation utilizing my entire dataset. I emphasize, I want to estimate my model performance for future predictions. Not the model performance in the past, where I had fewer data points available.