I have daily data for 2.5 years , but with more data points as 0, so when i excluded them in the cases which seems to be invalid. Can i use any other model than models used in time series or should i consider time series with missing values and proceed ? since they are quite a lot of missing values and am new to time series i dont know how to proceed with this. Any input is highly appreciated.
2
-
$\begingroup$ Please explain a bit more. What kind of daily data? You have many points that are 0 and you exclude them? Why do you think they are invalid? $\endgroup$– PaulSep 11, 2019 at 20:24
-
$\begingroup$ I have data regarding 200 departments , and their production time , alloted time and time which is not productive. but for few days daily data has not been recorded for some days there is no production. if i remove non productive days and non recorded days which are quite a lot, my prediction is not so good, if i clean them by replacing with mean or previous values,i still cant get better results. i want to know if there is better way to deal $\endgroup$– Meghana KanuriSep 12, 2019 at 12:32
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
1
First idea is to use, instead of an invalid data point, the last valid value. This is done in finance where some stocks are not quoted everyday (and when you want to publish the value of your fund you use the last available value for the stocks).
However, this can be an issue when you have too many missing values (especially for prediction).
You can try Bayesian data augmentation (works somewhat well with time series) but if you say that you have a lot of missing data you have to accept that anything that you do on rather poor quality data will be of limited quality.
-
$\begingroup$ I tried to add missing values with valid values, for prediction i have used auto arima and neural network for time series which gives me not very good prediction $\endgroup$ Sep 12, 2019 at 12:31
$\begingroup$
$\endgroup$
2
IMHO the existing answer is essentially correct, but perhaps it can be phrased more explicitly or clearly.
You have the option of trying to fill in the gaps before you run a time series model. Depending on how well you can predict the missing values, and how many there are, this can give a better result that trying to predict from the data with the missing values directly.
To fill in a gap, you can use the last known value. This is known as the "naive forecast". Similarly, you may want to interpolate the gap (essentially connect the points before and after the gap with a straight line). Or you may have more knowledge about your data that allows you to do a better forecast than this, or perhaps, knowing your data, you know that neither of these suggestions is any good.
For example, I don't know if it makes sense that if the production of a department on one day was at a certain level, that the production is probably similar the next day. Could be. Or not. I don't know the business.
After filling the gaps (or after choosing not to fill in any gaps), you will have to run a time series model. An ARIMA model would work. A standard recurrent neural net (like GRU or LSTM) would have trouble with gaps I think, but perhaps you know some tricks to avoid that.
I would also echo the last statement of the previous answer: If your data is bad, then you cannot expect to get great results, so perhaps you shouldn't be too surprised. There is also the question of whether the historical data fully explains the variations. For example, perhaps they're downsizing or upsizing some departments, and you wouldn't be able to predict that from the historical data.
-
$\begingroup$ Thank you for the explanation, Arima didn't work well with my data, but i have tried arima only on few departments , which i could identify seasonality but when i have 200 departments is there any quick way to identify seasonality. Neural network did a good job for certain departments $\endgroup$ Sep 16, 2019 at 11:17
-
$\begingroup$ You could use Prophet from Facebook. It’s super easy to use and it gives you weekly and seasonal patterns. $\endgroup$– PaulSep 16, 2019 at 13:08