When is it convenient dropping duplicates when performing Time Series Prediction?

Question

I have this Dataset.

 Head(df):

          ï..date   store item  qty    unit_price item_category 
 1220   2017-03-15    38  295   255      13.30            13 
 1453   2017-03-15    41   43   291      10.08            13 
 3181   2017-03-15    75  186   324      28.59            13 
 3541   2017-03-15    42  295   133      13.30            13 
 3624   2017-03-15    56  127   969      55.23            13 
 4657   2017-03-15    52   75   121      27.23            13 
 4702   2017-03-15    13  413    75      18.55            13 

There are clearly a lot of duplicates in the date column (since it depends on the stores and the items , that was expected), and since my task is to make a prediction and return predicted values in numeric of the qty variable - the target prediction variable "qty", represents the total quantities sold at the ITEM/DAY level- i think i can drop the duplicated dates (I'll be using the ARIMA model) and keep only the date , item and qty column for my prediction .

I wanted to know if my approach was right, any suggestion would be much appreciated , Thank you.

Answer 1

According to me, time-series ML Model are bit different than other routine ML models. As time-series model is based upon the sequence of previous values, dropping a Date may impact your solution.

Specially in this case where sale is definitely dependent upon the particular day. For eg consider store 38 is located in Area where on "2017-03-15" is holiday. So sale of store 38 will be comparatively high in comparison to other store and so on.

Rest you can try two different models one with Date and another without Date and compare the results as well.

Answer 2

You are throwing away a lot of valuable data by dropping "duplicates" because these observations are not really duplicates at all; your data appears to be a collection of distinct time series from a variety of different stores.

You can still use ARIMA here, but ARIMA is a univariate time series method. Some possible approaches off the top of my head;

1) Aggregate each individual time series/row by date (i.e. take the mean/median) so that you have an overall time series that is now independent of the store number. Depending on what you are trying to do, this might be okay, but you are still losing information here that could be useful.

2) Fit a single ARIMA/automated forecasting method to each individual time series you have in your dataset (so, fit a forecasting method to each individual store) and then use hierarchical time series methods to reconcile the fitted time series to the actual sum of all time series you observed in your dataset. Probably a more accurate method than 1), but more computation required.

3) Leave your dataset as you have shown, but change the date column to a time feature(s), example: 2017-03-15 is now 2017 + 74/365, or 2017-03-15 is now many columns; year = 2017, month = March, day = 15, week = 3, dayofWeek = Monday, or something like that. Now, treat the problem as a supervised learning (regression) problem. Incorporate lagged values of the target variable, or moving averages to (hopefully) capture short term trends. Indicate holidays/significant events on calendars, or promotional sales if you have access. Use store id's as a feature. In effect, you are trying to learn unique patterns specific to each store while also learning "global" patterns over all time series (if they exist). This is a more "machine learning" approach to time series forecasting and it works quite well for datasets that you appear to have; but it requires a lot of feature engineering (in general).

Answer 3

I'm trying to do something similar, and am still muddling through it myself, however the following might be helpful:

ML algorithms look at numbers as indicating a level of something, so when you have item numbers, you are probably going to need to one hot encode those, which will give you a very wide data set.

To associate the qty with the item, you might use the qty as the item column's value. Meaning you end up with something like this.

The stores could just be binary

 date           item295  item43  item186... store38  store41  store75
 2017-03-15     255      0       0          1        0        0
 2017-03-15     0        291     0          0        1        0
 2017-03-15     0        0       324        0        0        1

If you wanted to further reduce the number of rows, you might consider just doing a separate run for each store, and remove the store column(s) all together. Then all you would have left is one row per date, where each column represents an item, and the value is the qty.

Hope that gets you in the right direction.