2
$\begingroup$

I'm in the process of developing a new spark-based ARIMA(X) tool, and have reached the point where I need to know whether my coefficient estimates and forecasts are sensible. I can compare my results to R on the same data set, but, as my implementation is distributed while R's is in-memory, I think it's reasonable to assume that there will be some slight differences in our estimates. What I don't know is how different is too different for the coefficient estimates and forecasts? Is there a standard approach to evaluating to reasonableness of a new implementation of time series analytics?

$\endgroup$
  • $\begingroup$ Could you describe a bit about the data you are modeling? Is this a single time series, many different independent time series, hierarchical time series? Length of individual time series, frequency (or is it mixed). Is a single time series split and you are comparing the results? Is ensembling an option? $\endgroup$ – Paul Sep 21 '16 at 18:20
2
$\begingroup$

Is there a standard approach to evaluating [the] reasonableness of a new implementation of time series analytics?

Yes, there is. A way to validate your build-from-scratch is to simulate an ARIMAX timeseries from a data generating process (DGP) of your choice (see e.g. http://robjhyndman.com/hyndsight/arimax/).

Call your choice of DGP-parameters $\theta$, then:

  1. Draw a sample from the DGP.
  2. Estimate the model's parameters: $\hat{\theta}$. Make forecasts: $\hat{y}_{T+h|T}$.
  3. Assess $|| \hat{\theta} - \theta||_{\text{a metric}}$ and $||\hat{y}_{T+h|T} - y_{T+h}||_{\text{a metric}}$.
  4. Repeat 1 to 3 a couple of times, also for different DGPs and sample sizes.
  5. You either increase your confidence in the build, or find weak points that need improvement.

A sensible metric could be squared loss. Personally, I like to use the "eye-ball metric" for the forecast residuals, their distribution could be assessed by way of histogram.

|improve this answer|||||
$\endgroup$
  • $\begingroup$ Thanks, Jim! I suspected that's what the standard procedure was. Do you happen to have a publication reference I could use, or is the Hyndman text the best one here? $\endgroup$ – Kyle. Sep 21 '16 at 16:51
  • 1
    $\begingroup$ This is indeed the standard procedure. There are literally thousands of papers that include Monte-Carlo Simulation sections. For a very recent reference see Boot & Nibbering 2016 "Forecasting using Random Subspace Methods" Section 3. $\endgroup$ – Jim Sep 23 '16 at 20:11
0
$\begingroup$

What some people do is some kind of 'ordered' k-fold cross validation. (Check here, for instance.) You partition the data into $k$ partitions. Then, you fit the model on the partition 1, test on partition 2, get prediction error. Then, you fit the model on the union of partitions 1 and 2, test on partition 3, get prediction error. So on so forth, then average the prediction errors.

|improve this answer|||||
$\endgroup$
  • 1
    $\begingroup$ Yes, in a standard model development paradigm that's exactly what I'd do! However, because I literally built my ARIMAX approach from scratch, I'm looking to validate the code on data where one knows the sort of answer to expect. $\endgroup$ – Kyle. Sep 18 '16 at 1:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.