1
$\begingroup$

I'm trying to use LSTM (with Keras) for a time series problem. I would like predict the next value of the time series given its previous value. I'm using TimeseriesGenerator to create the training data as follows (setting length equal to 1 indicating the prediction is based on the previous value):

generator = TimeseriesGenerator(series, series, length=1, batch_size=10)

For the modeling I'm using the following:

model = Sequential()

model.add(LSTM(num_units, activation='relu', input_shape=(n_input, n_features)))

model.add(Dense(1))

model.compile(optimizer='adam', loss='mse')

model.fit_generator(generator, steps_per_epoch=len(generator)/n_batch, epochs=50)

for prediction:

yhat = model.predict(x_input, verbose=0)

when I set num_units = 1 (in the the first layer), the predicted values are much lower than the typical values in the time series (the typical values in the time series are 30-50 and the prediction is around 0.4). However, when I set num_units = 700, the predicted values become very close to the test values and the predictions seem to be overfitting. Why is that? What does the num_units in the first layer intuitively represent? If our input data is just one number, what does it mean to have a layer with 700 units? What is the intuition behind mapping a number to 700 neurons and what does one gain from it? In general, if in our time series, we're trying to make a prediction from the past n observations, how many units should be in the first LSTM layer?

$\endgroup$
1
$\begingroup$

num units is the number of hidden units in each time-step of the LSTM cell's representation of your data- you can visualize this as a several-layer-deep fully connected sequence of layers in which each layer also has a connection to a memory across the layers,even though that a analogy isn't 100% perfect. num units, then, is the number of units in each of those layers.

For your specific problem, and with length = 1, this reduces to a single layer- your model is not taking advantage of the memory capabilities of LSTM because there's simply nothing to remember beyond a single time step, because there's only a single time-step.

As to why it seems to perform better with a higher num units-- your model doesn't have enough expressive power at num units = 1 to describe the function it's trying to- there is too much information to encode in only a single unit. At num units = 700, I'd be very curious as to what the performance of the model is on out-of-sample data.

It's very likely that your network is simply over fitting, or memorizing your training data, with that much expressive power. If you can verify that that's happening- you'd see very good performance on the training data and poor performance on out-of-sample data- then you'll need to regularize the network in some way. Common ways to do this include implementing early stopping, introducing dropout layers, and/or reducing the number of hidden units in your layers.

Overall, if you don't have more than one time-step observation for a single entity, I would suggest that you change the LSTM layer to a simple fully connected layer to simplify the architecture. If you do or can get those sequential observations, then an LSTM layer might be a good solution for you.

$\endgroup$
3
  • $\begingroup$ Thanks Thomas. As I had mentioned the model performance is very good on the test data (what you called out-of-sample data). Any clues as what is the reason? $\endgroup$ Aug 21 '19 at 17:18
  • $\begingroup$ If it's good on out-of-sample data and your validation strategy is sound, I would diagnose you with a case of "has a good model" $\endgroup$ Aug 21 '19 at 19:10
  • $\begingroup$ OK! But it's too good to be true! The model follows all the fluctuations in the test data. It looks like over fitting but I'm not sure how it's over fitting given the fact that the test data and the training data are separate from each other and the model is only trained on the training data. So, given that the number of time steps is 1, the first layer basically maps a single number to 700 neurons. But what does that intuitively represent? $\endgroup$ Aug 22 '19 at 7:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.