LSTM number of units for first layer

Question

I'm trying to use LSTM (with Keras) for a time series problem. I would like predict the next value of the time series given its previous value. I'm using TimeseriesGenerator to create the training data as follows (setting length equal to 1 indicating the prediction is based on the previous value):

generator = TimeseriesGenerator(series, series, length=1, batch_size=10)

For the modeling I'm using the following:

model = Sequential()

model.add(LSTM(num_units, activation='relu', input_shape=(n_input, n_features)))

model.add(Dense(1))

model.compile(optimizer='adam', loss='mse')

model.fit_generator(generator, steps_per_epoch=len(generator)/n_batch, epochs=50)

for prediction:

yhat = model.predict(x_input, verbose=0)

when I set num_units = 1 (in the the first layer), the predicted values are much lower than the typical values in the time series (the typical values in the time series are 30-50 and the prediction is around 0.4). However, when I set num_units = 700, the predicted values become very close to the test values and the predictions seem to be overfitting. Why is that? What does the num_units in the first layer intuitively represent? If our input data is just one number, what does it mean to have a layer with 700 units? What is the intuition behind mapping a number to 700 neurons and what does one gain from it? In general, if in our time series, we're trying to make a prediction from the past n observations, how many units should be in the first LSTM layer?

Answer 1

num units is the number of hidden units in each time-step of the LSTM cell's representation of your data- you can visualize this as a several-layer-deep fully connected sequence of layers in which each layer also has a connection to a memory across the layers,even though that a analogy isn't 100% perfect. num units, then, is the number of units in each of those layers.

For your specific problem, and with length = 1, this reduces to a single layer- your model is not taking advantage of the memory capabilities of LSTM because there's simply nothing to remember beyond a single time step, because there's only a single time-step.

As to why it seems to perform better with a higher num units-- your model doesn't have enough expressive power at num units = 1 to describe the function it's trying to- there is too much information to encode in only a single unit. At num units = 700, I'd be very curious as to what the performance of the model is on out-of-sample data.

It's very likely that your network is simply over fitting, or memorizing your training data, with that much expressive power. If you can verify that that's happening- you'd see very good performance on the training data and poor performance on out-of-sample data- then you'll need to regularize the network in some way. Common ways to do this include implementing early stopping, introducing dropout layers, and/or reducing the number of hidden units in your layers.

Overall, if you don't have more than one time-step observation for a single entity, I would suggest that you change the LSTM layer to a simple fully connected layer to simplify the architecture. If you do or can get those sequential observations, then an LSTM layer might be a good solution for you.