1
$\begingroup$

Apologies for the awkward title, but I hope to be able to regain your confidence.

Let's start with the final output I got, so at least you can understand why I'm not happy/concerned about the outcome.

enter image description here

The graph shows the past 2 years website traffic. I was interested to understand whether my learnings could have been applicable to my main profession. I truly believe this is the case, but the results I got aren't very encouraging.

The plot has been obtained running the data past an ARIMA model after different normalisations occurred (e.g. differencing, log, etc.) The best ARIMA model was found to be 11, 1, 11. This is supported from the below:

enter image description here

You may be tempted to say P = 15, but in fact, after several computations, the best RSME was given by the above. See below (though I'm not sure that RSME can be really applied here)

enter image description here

I understand that almost 69% of confidence is far from being perfect, but I would have in some case expected to return something better than the above output.

Do you have any ideas of what I could have done wrong, what I may improve and how? Bear in mind I'm learning, so please excuse my limited understanding on complex terminology. The plainer you go, the best chance for me to understand.

Thanks for your support.

EDIT:

Thanks to all comments below, I made some progress. I was also able to get through the SARIMA and counter verify the data with the new data acquired since I started this learning project.

See below the new plot.

enter image description here

Although a perfectly matching fit with the "training" data, when it comes to the forecast, the data hardly correspond. Given my in-expertise, is the above plot something that you would deem normal? Or Do I need to re-work a lot more to get anything better?

$\endgroup$
1
$\begingroup$

I’m by no means an expert in ARIMA models but I have experimented a bit with them so I can give you a list of some things that I found to improve my models.

  1. I did a grid search of different ARIMA model parameters to find the best model

  2. I also tried some methods to ensemble ARIMA models. For example, averaging the best three ARIMA models gave a better overall score than just one.

  3. Depending on the problem you can play with what level the data is aggregated. I was interested in predicting weekly sales and so I made a model to predict daily sales and then aggregated the predictions to weekly sales. This improved the accuracy dramatically.

Looking at your autocorrelation there appear to be regular weekly patterns. Trying a SARIMA model with a 7 day cycle may result in better scores, I.e. there is a 7 day seasonality.

Finally, I also tried other model types, like facebook’s prophet package or converting it to a regression problem and using weekday, week number, month and holiday days as features in a random forest.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Hey @ThomasG, thanks for feeding in. Can you give more details on point 2)? what exactly did you averaged? Was it the results of the predictions? Did you used some kind of train/test split or did you go straight with the whole bucket into prediction? $\endgroup$ – Andrea Moro May 25 at 20:16
1
$\begingroup$

Seems you have a seasonal correlation - every seventh day. You can look at the SARIMAX models in statsmodels to try to compensate for that. It adds a fourth component to your ARIMA, so on top of p, d and q, you get a fourth term, s. Here is a walkthrough on how to work with that.

Using an ensemble of models does help to remove noise a little further, as mentioned by @ThomasG. This is used in many modelling situations to eek out the last few accuracy points, but won't solve an underlying modelling issue.

Maybe you could create a stratified train/validation/test split, such that the model is trained (and then evaluated) on all kinds of web traffic throughput: increasing, decreasing, stable. Stratification really just means that those three phases are equally represented in the splits you create.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Surely there is a weekly seasonality. This is almost expected in any e-commerce website after all with week-end deepening a bit. I was however convinced that differencing would have solve this problem? $\endgroup$ – Andrea Moro May 26 at 7:19
  • $\begingroup$ It depends; how did you perform differencing? $\endgroup$ – n1k31t4 May 26 at 7:43
  • $\begingroup$ I did something very simple using the .diff() method of Panda, the only one I was thought to date. Are there any other ways of doing it? $\endgroup$ – Andrea Moro May 26 at 15:15
  • 1
    $\begingroup$ Perfoming single-step differencing alone (i.e. day to day) doesn't solve the seasonality, because you really have a kind of state change (the seasonality). Think what differencing is doing... when you do e.g Saturday - Friday and also Monday - Sunday, you will create huge swings in the opposite directions. Differencing simply helps to de-trend. You can just repeat differencing as you did before, using the period (7 days), so weekends are subtracted. Experiment with this and also the s parameter of SARIMAX as mentioned above. $\endgroup$ – n1k31t4 May 26 at 15:27
  • $\begingroup$ When you say repeat the differencing, you mean to apply another round of .diff() on top of the same dataset, or start fresh and apply it with a periods=7 do simulate the weekend? $\endgroup$ – Andrea Moro May 26 at 18:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.