5
$\begingroup$

I trained a LSTM network on a time series dataset. Predictions seem to follow the dataset. In fact, they are nearly a right shifted form of real values. Thus, in my opinion, it doesn't provide any predictive capability since only shifting a signal in time results in low RMSE but is not useful.

How to properly evaluate a time series model?

$\endgroup$
5
  • 2
    $\begingroup$ your testPred plot doesn't start at zero. Are you sure you're plotting it right? $\endgroup$ Mar 2, 2017 at 16:11
  • $\begingroup$ @MohammadAthar, testPred is the forecast. There needs to be some amount of data before making a prediction, which is why testPred does not start at 0. $\endgroup$
    – Hobbes
    May 1, 2017 at 15:22
  • $\begingroup$ @Horacet not sure why you're singling me out for this info, since I just asked if the data are plotted right $\endgroup$ May 1, 2017 at 20:00
  • 1
    $\begingroup$ @MohammadAthar I meant to address the author of the post. Sorry. $\endgroup$
    – horaceT
    May 1, 2017 at 20:09
  • $\begingroup$ @Mustafa You have to provide a lot more details about your model and data before anyone could help you. First, is the response just an univariate time series? what's your predictors that got fed into the LSTM? is it just $y_t$ lagged by a few time steps? what's the LSTM arch? $\endgroup$
    – horaceT
    May 1, 2017 at 20:11

3 Answers 3

3
$\begingroup$

The best summary on evaluating time series forecast is probably explained in detail on Rob Hyndman's site. I typically use the mean absolute percentage error which is baked in Keras. However, what I found in a different setting is that the MAPE prevents the neural network from converging if combined with the Adam optmization. I had much better success with the rooted mean square error (RMSE). Since you have poor experience with that maybe you could use the symmetric MAPE.

$\endgroup$
5
  • 1
    $\begingroup$ My problem with examples of time series forecasting is that they never seem to actually forecast. For example, on Rob Hyndman's site there is the example of forecasting beer production. The forecasted results are essentially just the last 'season'. This could be easily predicted by eye. What advantage is there to modelling something that is very clear by looking at the data itself? I'm interested in this generally, not saying the answer is bad. $\endgroup$
    – Hobbes
    May 1, 2017 at 15:44
  • $\begingroup$ @Stereo RH has done a lot of great works on time series forecasting, but when it comes to forecasting with state-of-the-art deep learning models, such as LSTM recurrent neural nets, his techniques and approaches aren't very relevant. Whether MAPE, MAD, RMSE, or MSE, it all depends on how well behave the individual data points are. There is absolutely no general rule here. $\endgroup$
    – horaceT
    May 1, 2017 at 20:05
  • 1
    $\begingroup$ @Hobbes I up-voted your comment. Most time series models have little forecasting power. They just spit out either 1) the last value, 2) the mean of the time points corresponding to the historic periodicity. $\endgroup$
    – horaceT
    May 1, 2017 at 20:08
  • $\begingroup$ @horaceT, Thanks for commenting. I've had a few time series problems so far that I have really struggled with and haven't found reliable solutions yet. I'm always interested in time series questions though. $\endgroup$
    – Hobbes
    May 1, 2017 at 21:03
  • $\begingroup$ @horaceT I fully agree with you that it very much depends on the data points what error measure is most useful. I have seen cases where my team decided to penalize predictions with a wrong sign heavier. Just to show that it is essential to evaluate case by case what a 'good' measure is. $\endgroup$
    – Stereo
    May 2, 2017 at 8:54
2
$\begingroup$

If you are evaluating point estimates (i.e. single number estimates) then you are well advised to use a proper scoring rule. Some metrics elicit "honest" estimates whereas others do not. A characterization of proper scoring rules was provided by Savage. Most people go with squared error, logarithmic.

Contest sites like M-Competitions or microprediction have moved towards assessing distributional estimates, not point estimates. You can also find decent benchmarks, at least for univariate prediction, by searching for "time-series Elo ratings".

$\endgroup$
1
$\begingroup$

As JQ Veenstra has pointed out your method of evaluation depends a lot on the particular type of time series model that you are estimating. Have a look at the following points.

Usually you should have a set of residuals in your model that are uncorrelated. You can test that. You can test the forecasting ability of your model by starting with a subset of the data recursively estimate the model and look at the errors when forecasting each re-estimated model. For general guidance on forecasting I would recommend Granger, Clive W. J. and Paul Newbold (1986) -Forecasting Economic Time Series - Academic Press 1986 which is a bit dated but covers well many aspects of forecast evaluation. Elliott, G. and Timmermann, A (2016). Economic Forecasting, Princeton University Press is, perhaps a little mathematical but provides a comprehensive coverage of forecasting. The references to specific areas in this may give you more guidance on the evaluation of specific forecalting methods.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.