Time series modelling

Question

I have daily data for 2.5 years , but with more data points as 0, so when i excluded them in the cases which seems to be invalid. Can i use any other model than models used in time series or should i consider time series with missing values and proceed ? since they are quite a lot of missing values and am new to time series i dont know how to proceed with this. Any input is highly appreciated.

Answer 1

First idea is to use, instead of an invalid data point, the last valid value. This is done in finance where some stocks are not quoted everyday (and when you want to publish the value of your fund you use the last available value for the stocks).

However, this can be an issue when you have too many missing values (especially for prediction).

You can try Bayesian data augmentation (works somewhat well with time series) but if you say that you have a lot of missing data you have to accept that anything that you do on rather poor quality data will be of limited quality.

Answer 2

IMHO the existing answer is essentially correct, but perhaps it can be phrased more explicitly or clearly.

You have the option of trying to fill in the gaps before you run a time series model. Depending on how well you can predict the missing values, and how many there are, this can give a better result that trying to predict from the data with the missing values directly.

To fill in a gap, you can use the last known value. This is known as the "naive forecast". Similarly, you may want to interpolate the gap (essentially connect the points before and after the gap with a straight line). Or you may have more knowledge about your data that allows you to do a better forecast than this, or perhaps, knowing your data, you know that neither of these suggestions is any good.

For example, I don't know if it makes sense that if the production of a department on one day was at a certain level, that the production is probably similar the next day. Could be. Or not. I don't know the business.

After filling the gaps (or after choosing not to fill in any gaps), you will have to run a time series model. An ARIMA model would work. A standard recurrent neural net (like GRU or LSTM) would have trouble with gaps I think, but perhaps you know some tricks to avoid that.

I would also echo the last statement of the previous answer: If your data is bad, then you cannot expect to get great results, so perhaps you shouldn't be too surprised. There is also the question of whether the historical data fully explains the variations. For example, perhaps they're downsizing or upsizing some departments, and you wouldn't be able to predict that from the historical data.