Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
I trained a LSTM network on a time series dataset. Predictions seem to follow the dataset. In fact, they are nearly a right shifted form of real values. Thus, in my opinion, it doesn't provide any predictive capability since only shifting a signal in time results in low RMSE but is not useful.
How to properly evaluate a time series model? 
(source: image.ibb.co)
The best summary on evaluating time series forecast is probably explained in detail on Rob Hyndman's site. I typically use the mean absolute percentage error which is baked in Keras. However, what I found in a different setting is that the MAPE prevents the neural network from converging if combined with the Adam optmization. I had much better success with the rooted mean square error (RMSE). Since you have poor experience with that maybe you could use the symmetric MAPE.
If you are evaluating point estimates (i.e. single number estimates) then you are well advised to use a proper scoring rule. Some metrics elicit "honest" estimates whereas others do not. A characterization of proper scoring rules was provided by Savage. Most people go with squared error, logarithmic.
Contest sites like M-Competitions or microprediction have moved towards assessing distributional estimates, not point estimates. You can also find decent benchmarks, at least for univariate prediction, by searching for "time-series Elo ratings".
As JQ Veenstra has pointed out your method of evaluation depends a lot on the particular type of time series model that you are estimating. Have a look at the following points.
Usually you should have a set of residuals in your model that are uncorrelated. You can test that. You can test the forecasting ability of your model by starting with a subset of the data recursively estimate the model and look at the errors when forecasting each re-estimated model. For general guidance on forecasting I would recommend Granger, Clive W. J. and Paul Newbold (1986) -Forecasting Economic Time Series - Academic Press 1986 which is a bit dated but covers well many aspects of forecast evaluation. Elliott, G. and Timmermann, A (2016). Economic Forecasting, Princeton University Press is, perhaps a little mathematical but provides a comprehensive coverage of forecasting. The references to specific areas in this may give you more guidance on the evaluation of specific forecalting methods.
If you evaluate a time series model, you normally calculate naive predictions (e.g. predictions without any model) and compare those values with your model results. In this way you can see if your model has some prediction power. As naive predictions you can use a mean value or the last observation before your target value.
