Train and test split for regression

Question

I’m trying to build a regression model that estimates the amount of sales of a beer product on a given day based on the prices of the product and competitors, the weather, the season and the day of week of that specific day

My question is how to split the data into train and test

Because I pretend to use the model to make a prediction for a future day, I think I could split the data so that the test set is composed of those observations with recent dates and the train test with past dates

This make sense or I should split at random?

Answer 1

Try to do the Time based splitting. Ex: If you have 1 year data, first 1-8 months for Train, 9-10 for Validation, 11-12 for Testing. After this, a better option is to go with some sliding/moving window method to create the ML problem like using past 10-15 days sales plus static and dynamic features of the day predict the sales of the day or next day.

Answer 2

I think this is a multivariate time series problem, for which simple regression might not be good choice as you want to predict future values. Although as far as the question of splitting dataset is concerned, you should split the data as:

data = train + validation + test
        |            |         |
        V            V         V
     (2 years)  (2 months)  (2 months)

1. Parse date column.
2. Convert date column as index for the dataframe.
3. The data must be split according to time/date.

For further information on Multivariate Time Series you can refer this article:

https://www.analyticsvidhya.com/blog/2018/09/multivariate-time-series-guide-forecasting-modeling-python-codes/

Hope this could help.

Answer 3

You can use time-series split from sklearn here