Time series forecasting: prediction and forecast far from the reality

Question

Apologies for the awkward title, but I hope to be able to regain your confidence.

Let's start with the final output I got, so at least you can understand why I'm not happy/concerned about the outcome.

The graph shows the past 2 years website traffic. I was interested to understand whether my learnings could have been applicable to my main profession. I truly believe this is the case, but the results I got aren't very encouraging.

The plot has been obtained running the data past an ARIMA model after different normalisations occurred (e.g. differencing, log, etc.) The best ARIMA model was found to be 11, 1, 11. This is supported from the below:

You may be tempted to say P = 15, but in fact, after several computations, the best RSME was given by the above. See below (though I'm not sure that RSME can be really applied here)

I understand that almost 69% of confidence is far from being perfect, but I would have in some case expected to return something better than the above output.

Do you have any ideas of what I could have done wrong, what I may improve and how? Bear in mind I'm learning, so please excuse my limited understanding on complex terminology. The plainer you go, the best chance for me to understand.

Thanks for your support.

EDIT:

Thanks to all comments below, I made some progress. I was also able to get through the SARIMA and counter verify the data with the new data acquired since I started this learning project.

See below the new plot.

Although a perfectly matching fit with the "training" data, when it comes to the forecast, the data hardly correspond. Given my in-expertise, is the above plot something that you would deem normal? Or Do I need to re-work a lot more to get anything better?

Answer 1

I’m by no means an expert in ARIMA models but I have experimented a bit with them so I can give you a list of some things that I found to improve my models.

1. I did a grid search of different ARIMA model parameters to find the best model

2. I also tried some methods to ensemble ARIMA models. For example, averaging the best three ARIMA models gave a better overall score than just one.

3. Depending on the problem you can play with what level the data is aggregated. I was interested in predicting weekly sales and so I made a model to predict daily sales and then aggregated the predictions to weekly sales. This improved the accuracy dramatically.

Looking at your autocorrelation there appear to be regular weekly patterns. Trying a SARIMA model with a 7 day cycle may result in better scores, I.e. there is a 7 day seasonality.

Finally, I also tried other model types, like facebook’s prophet package or converting it to a regression problem and using weekday, week number, month and holiday days as features in a random forest.

Answer 2

Seems you have a seasonal correlation - every seventh day. You can look at the SARIMAX models in statsmodels to try to compensate for that. It adds a fourth component to your ARIMA, so on top of p, d and q, you get a fourth term, s. Here is a walkthrough on how to work with that.

Using an ensemble of models does help to remove noise a little further, as mentioned by @ThomasG. This is used in many modelling situations to eek out the last few accuracy points, but won't solve an underlying modelling issue.

Maybe you could create a stratified train/validation/test split, such that the model is trained (and then evaluated) on all kinds of web traffic throughput: increasing, decreasing, stable. Stratification really just means that those three phases are equally represented in the splits you create.