0
$\begingroup$

I am trying to tune the hyperparameters of a LSTM I have to do time series forecasting. I have noticed that my validation accuracy is always very close to my training accuracy. I am not sure whether or not this is good or bad or what it implies in general?

At the moment I have kept all the hyperparameters the same and have only varied the number of units in the LSTM layer from [1, 5, 26].

I expected 26 units to give me good results and have since added units 1 and 5 to help me investigate. I was also expecting to see that my validation accuracies are worse than my training accuracies but this does not seem to be the case. My validation accuracies track the training accuracies very well. Is this something to be concerned about?

The plot below shows the average loss, average training and validation accuracies. training validation comparison

You can see from the plot that each of the units training and validation accuracies stay very close together. Why is this? Does this mean that in general my model should generalise well on unseen data as there isn't much of a discrepancy?

Obviously the accuracies of the model are not very good yet in general and it requires some tuning but to do the tuning process I was more expecting to see some differences between the training and validation data and not that they would stay largely similar to each other.

EDIT:

Further information as requested. The hyperparameters used for each the model can be seen in the legend of the plots. Here is an updated plot which shows a greater variation in accuracy from varying the number of neurons and batch size:

training validation accuracies

From the plot it seems as if the model accuracy converges the quickest with a batch size of 1 and 39 neurons. However it is characteristic amongst all that the training and validation accuracies track each other closely. I hadn't expected that varying two hyperparameters independently would lead to a consistent result like that.

In this problem I am working with, the model is to provide forecasts for one time series. I am using it as a 'toy' problem to try and learn what kind of models work well for my particular problem.

I have 315 data points in total for the time series. I have left the last 52 out as a hold out test set which I haven't ever looked at to this point. 52 points as I am making weekly predictions. I then left the next last 52 points out to use as a validation set. Due to the time dependency I am unable to use a validation method such as KFold cross validation. Therefore I am using something called rolling origin analysis which is explained well here (see under predictive performance). this simulates well how the model would be used in practice and my accuracies are measurements using a modified sMAPE formula for a multi-step forecast.

Essentially what this means is that I train (in this case) 27 separate models. I have a forecast horizon of 26, so what I do is take my train data and take the first 26 points of my validation set.

I train my first model on the train data (which is 131 samples) and this is validated against 1 sample.

I train the next model which uses the same training data except it is shifted one along so the very first point from train is dropped and the first point from the validation set is appended to the end of the train set. The validation set is then also shifted along by one. As a quick example (numbers represent indices of the time series):

Model1 train: [1, 2, 3, 4, 5], val: [6, 7, 8]

Model2 train: [2, 3, 4, 5, 6], val: [7, 8, 9]

The average accuracy shown in the plots is therefore the average of the accuracies that each model computed against its own validation set.

So each model is trained on the same amount of data and each model is completely fitted anew, that is Model1 doesn't share any information with Model2.

The accuracies are computed in this way because I do not have much data and this is the best way I have found that allows me to get more than two validation sets and more than two test sets.

$\endgroup$
4
$\begingroup$

The fact that the training accuracy and the validation accuracy are close it is nothing to be concerned about. As you mention, it means your model are generalizing well.

The thing to be worried about is the low training accuracy. It seems that your model is under-fitting the data but with low variance. This is the typical scenario of a model with high bias and low variance.

The bias is the error (whatever you measure this error) in your training data. It tells you about how well the model is forecasting with seen data. High bias means poor performance (under-fitting). The variance is the difference in performance between the training data and the validation data. If the differences are high (far better performance in training than in validation) the model has high variance (over-fitting).

Typically, decreasing the bias implies increasing the variance (bias-variance trade-off) then, probably, as you get better results in training, the validationa ccuracy may decrease. But, if the model is well designed, you may end up with low bias and low variance.

$\endgroup$
  • $\begingroup$ Ok thanks. The plot above was just produced quickly to demonstrate the behaviour I am getting. With a smaller batch size (but obviously longer run time) I am getting higher accuracies ~80%, which is performant compared to other models used for the same problem. It does however still feature the same characteristic that the training and validation accuracy are similar. I was concerned because it seems that regardless of what hyperparameters I change, I get this behaviour whatever. Is that normal or could that suggest a problem? $\endgroup$ – Aesir Dec 14 '18 at 11:49
  • 1
    $\begingroup$ It's difficult to say if we do not have more details about the model the data, the train-val splitting... If you want further insight , it is worth you add more details to the question $\endgroup$ – ignatius Dec 14 '18 at 12:05
  • $\begingroup$ Sure ok I will do that, thanks for taking a look so far. $\endgroup$ – Aesir Dec 14 '18 at 12:06
  • $\begingroup$ You're very welcome $\endgroup$ – ignatius Dec 14 '18 at 12:07
  • $\begingroup$ I added extra information as requested about how the problem is set up and another plot which demonstrates hopefully a bit better why I am confused. If there is anything else more specific you think would help to add, please just ask! $\endgroup$ – Aesir Dec 14 '18 at 12:25
1
$\begingroup$

1

It seems to me that you have very few data points... Your are using Deep Learning, which is a data hungry technique and 315 points are a really small data-set...

2

This is a toy example, where does your data come from? If you created it (for example, a sine) the problem may be easy... I expect you are not trying to forecast a random walk: in time series, looking only at the MAE is not always the best idea. In a random walk, you may be forecasting a sample with the same value as the previous one, and the error still becomes small... Try to plot the true curves and the predicted ones...

3

For the first epochs, the validation accuracy is higher than the one in training. This is quite weird... It may happen if your model is using dropout and you are not disabling it in the validation stage.

4

Once again, having the same accuracy in train and test set is not a bad symptom. You have to evaluate your results with respect to your data, your model, your training technique and so on...

Comments:

Mos of the times I encountered that the problem with a Deep Learning model comes from a bad data handling procedure (data gathering, data cleaning, data splitting, data processing and so on) and with misleading interpretations of the results, and not from the model itself. Please don not take it wrong, I'm not saying is necessary your case...

$\endgroup$
  • $\begingroup$ The data is a real time series not artificial it is unfortunately limited in size to the number of points that I have. I don't think that it is the best solution to the problem but right now I am trying to pick a suitable model from many (ARIMA, ETS, LSTMs etc) and at the moment the validation accuracy is comparable to that of simpler models. So I'd like to just confirm the model is working correctly before I evaluate it on the test set, so that I can correctly compare how good it is w.r.t the other models I have tried. $\endgroup$ – Aesir Dec 14 '18 at 13:40
  • $\begingroup$ I also think it is weird my validation accuracy typically is better than my training accuracy... I have built the model using keras and it is quite simple. To my knowledge it doesn't use any dropout (or at least I have not configured any) $\endgroup$ – Aesir Dec 14 '18 at 13:42
  • $\begingroup$ The main problem for me is that you are using a deep learning model with very few data... My personal opinion is that ittle can be inferred about the model with this small data-set. For me, is not worth working on this model with this data... I strongly recommend you try it with some other data-set $\endgroup$ – ignatius Dec 14 '18 at 13:48
  • 1
    $\begingroup$ Not necessary... but with a small data-set, I feel it is hard to interpret results $\endgroup$ – ignatius Dec 14 '18 at 14:06
  • 1
    $\begingroup$ Yes, for sure. It is the first thing I said... I won't be specially concerned about the fact your model is generalizing well... But you still get the feel that something is wrong, maybe because you don't expected having good accuracy so early. Your model may be ok! But you want to relate the fact that hyper-parameter tuning does not affect the results, but indeed, they affect... as you can see when you increase the number of neurons... $\endgroup$ – ignatius Dec 14 '18 at 14:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.