# ARIMA(X) Validation

I'm in the process of developing a new spark-based ARIMA(X) tool, and have reached the point where I need to know whether my coefficient estimates and forecasts are sensible. I can compare my results to R on the same data set, but, as my implementation is distributed while R's is in-memory, I think it's reasonable to assume that there will be some slight differences in our estimates. What I don't know is how different is too different for the coefficient estimates and forecasts? Is there a standard approach to evaluating to reasonableness of a new implementation of time series analytics?

• Could you describe a bit about the data you are modeling? Is this a single time series, many different independent time series, hierarchical time series? Length of individual time series, frequency (or is it mixed). Is a single time series split and you are comparing the results? Is ensembling an option?
– Paul
Commented Sep 21, 2016 at 18:20

Is there a standard approach to evaluating [the] reasonableness of a new implementation of time series analytics?

Yes, there is. A way to validate your build-from-scratch is to simulate an ARIMAX timeseries from a data generating process (DGP) of your choice (see e.g. http://robjhyndman.com/hyndsight/arimax/).

Call your choice of DGP-parameters $\theta$, then:

1. Draw a sample from the DGP.
2. Estimate the model's parameters: $\hat{\theta}$. Make forecasts: $\hat{y}_{T+h|T}$.
3. Assess $|| \hat{\theta} - \theta||_{\text{a metric}}$ and $||\hat{y}_{T+h|T} - y_{T+h}||_{\text{a metric}}$.
4. Repeat 1 to 3 a couple of times, also for different DGPs and sample sizes.
5. You either increase your confidence in the build, or find weak points that need improvement.

A sensible metric could be squared loss. Personally, I like to use the "eye-ball metric" for the forecast residuals, their distribution could be assessed by way of histogram.

• Thanks, Jim! I suspected that's what the standard procedure was. Do you happen to have a publication reference I could use, or is the Hyndman text the best one here? Commented Sep 21, 2016 at 16:51
• This is indeed the standard procedure. There are literally thousands of papers that include Monte-Carlo Simulation sections. For a very recent reference see Boot & Nibbering 2016 "Forecasting using Random Subspace Methods" Section 3.
– Jim
Commented Sep 23, 2016 at 20:11

What some people do is some kind of 'ordered' k-fold cross validation. (Check here, for instance.) You partition the data into $k$ partitions. Then, you fit the model on the partition 1, test on partition 2, get prediction error. Then, you fit the model on the union of partitions 1 and 2, test on partition 3, get prediction error. So on so forth, then average the prediction errors.

• Yes, in a standard model development paradigm that's exactly what I'd do! However, because I literally built my ARIMAX approach from scratch, I'm looking to validate the code on data where one knows the sort of answer to expect. Commented Sep 18, 2016 at 1:40