I make this sec2sec NN model for the purposes of learning:

model = Sequential()
model.add(LSTM(100, activation="relu", input_shape=(3,1)))
model.add(LSTM(100, activation="relu", return_sequences=True))
model.compile(optimizer="adam", loss="mse")

It was trained on this sequence: [10, 20, 30, 40, 50, 60, 70, 80, 90] for using like this

x_test = array([70, 80, 90])
x_test = x_test.reshape(1, 3, 1)
model.predict(x_test, verbose=0)

I can not understand whats happen with input data inside all 100 LSTM cells when I invoke model.predict().

Explain me please what all this 100 units do if x_test array goes into one unit input.


The test data is a 3 step sequence data i.e. 70,80,90.

Following will be the flow -

  • 70 will reach the 1st LSTM layer i.e. to the all 100 in parallel
  • It will also come back by the Recurrent weight to be multiplied with the next seq.
  • 80 will reach and add up with the Recurrent of 70.
  • Similarly, 90 will pass through the 1st LSTM layer.
  • All the seq_step is held at 1st layer because return_sequences=Flase
  • After the last seq_step, the output will reach the next layer
  • RepeatVector will duplicate the vector, so the single output will become 2 step input for the next LSTM layer.
  • When the first of the duplicate will reach the LSTM, it will behave exactly like the previous LSTM layer. In addition, it will pass the output to the next layer too because return_sequences=True here.
  • Same thing will repeat for the 2nd duplicate vector.
  • The role of TimeDistributedlayer is to wrap the two incoming data as two steps and hence you will receive a (2,) output.

How vectors look across the Recurrent Layers

  • This is intuitive for a single Neuron/Single Feature i.e. as depicted below

$\hspace{5cm}$ enter image description here

  • For a multi-feature, multiple Neuron, it goes like below
    Your case has a single feature for each step and 100 Neurons. The below depiction has 3 features and 4 Neurons. Features behave the way it does in a simple feedforward network. The only new information here is that recurrent of each Neuron goes back to each of the 100. enter image description here

    $\hspace{7cm}$ Image credit - SO Answer
  • $\begingroup$ So you say that each time step go to each lstm cell. But why almost every lstm article show a picture where X_{t-1} goes to LSTM_1, X_t goes to LSTM_2, X_{t+1} goes to LSTM_3? hsto.org/web/67b/04f/73b/67b04f73b4c34ba38edfa207e09de07c.png $\endgroup$ – Kroll May 7 at 16:11
  • $\begingroup$ These explanations are for single Neurons unrolled across time. $\endgroup$ – 10xAI May 7 at 18:50
  • $\begingroup$ Why my case is not? What the difference? $\endgroup$ – Kroll May 7 at 22:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.