3
$\begingroup$

My question is on the number of units in an LSTM cell.

I've come across the following example which is a model for predicting a value in a series based on its 2 lag observations. I'm wondering why the number of units in the LSTM cell is 100 which is much higher than the number of features. In general, wouldn't be more logical to set the number of units to the number of input features? What is the advantage of having a number of units higher than the number of features? And what does it intuitively mean?

My code:

series = array([1, 2, 3, 4, 5, 6, 7, 8, 9, 10])

n_features = 1

series = series.reshape((len(series), n_features))

n_input = 2

generator = TimeseriesGenerator(series, series, length=n_input, batch_size=8)

model = Sequential()

model.add(LSTM(100, activation='relu', input_shape=(n_input, n_features)))

model.add(Dense(1))

model.compile(optimizer='adam', loss='mse')
$\endgroup$

1 Answer 1

1
$\begingroup$

There are very few resources that justify number of cells proportional to input. The intuition though is clear from colah's blog. The longer the sequence you want to model, the more number of cells you need to have in your layer. For e.g. if you are using the LSTM to model time series data with a window of 100 data points then using just 10 cells might not be optimal. The goal of any RNN (LSTM/GRU) is to be able to encode the entire sequence into a final hidden state which it can then pass on to the next layer. So even if you look at the older NMT models (sequence to seq for language translation), at a very basic level, it had a layer of an lstm encoding all info from the sequence that was passed to it and then using it to kick start the decoding. So basically, if the number of units are relatively lesser, the ability to encode all the info MIGHT not be optimal.

Having said that, we regularly use

a) multiples of 32 (https://svail.github.io/rnn_perf/) so inspite of their explanation being vague at best, the way to look at it is that when u declare an output size of say, 100, the RNN will generate square matrices of 100 x 100 with weights in them (that will be adjusted during back prop to give you a final model) and that matrix multiplications of such a matrix will be unwieldy as opposed to a matrix thats a multiple of 32 ( this is totally my intuition again, please correct, if im mistaken)

b) also if you use more than a certain number of hidden units, you will end up with the vanishing gradient problem (exploding gradients typically dont occur due to relu activation functions that keep activations between 0 and 1). Hence you are better off going "deeper" than "wider" , where you can stack 2-3 layers of 128 hidden cells "vertically", instead of using 512 hidden cells in one layer (this , i have actually experienced in almost all forms of experiments with text as well as image data)

im hoping this clarified matters somewhat

$\endgroup$
3
  • $\begingroup$ Thanks. I'm still not clear though. The input to LSTM has the shape (batch_size, time_steps, number_features) and units is the number of output units. So, in the example I gave you, there are 2 time steps and 1 input feature whereas the output is 100. I don't understand what does it mean when a 1 dimensional feature is becoming 100 dimensional. It would be more intuitive to have the number of units to be smaller than the number of features as in for example: medium.com/@shivajbd/… (number of features=10 and units=3) $\endgroup$ Aug 11, 2019 at 19:41
  • $\begingroup$ sure; typically to predict a series you need a window of observations. For e.g. if i want to look at last "n" days and predict today, you are looking at a typical moving average, correct ? in this case you don't need 100 at all, anything > 1 will do. Why ">1" u ask ? because having just 1 hidden unit is basically a linear regressor. Most pattern recognition problems like to model some form of a polynomial function (quadratic, for e.g.). But then again if your data is linear, then there's no use for an AI approach as a simple statistical model should work no ? Continuing in the comment below .. $\endgroup$ Aug 12, 2019 at 6:06
  • $\begingroup$ So a simple solution would be to plot the series you are modeling and if it displays a pattern that can't be fit using a regular linear, quadratic / any polynomial function then use the RNN approach. If the pattern is too irregular, it would help to start with a good number of hidden units and then increase or decrease based on how well the model fits your training and validation sets. Whats a good number ? could be anything but definitely more than 1 :) ..hope it clarifies a bit more $\endgroup$ Aug 12, 2019 at 6:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.