How an input data flows through an lstm layer cells?

Question

I make this sec2sec NN model for the purposes of learning:

model = Sequential()
model.add(LSTM(100, activation="relu", input_shape=(3,1)))
model.add(RepeatVector(2))
model.add(LSTM(100, activation="relu", return_sequences=True))
model.add(TimeDistributed(Dense(1)))
model.compile(optimizer="adam", loss="mse")

It was trained on this sequence: [10, 20, 30, 40, 50, 60, 70, 80, 90] for using like this

x_test = array([70, 80, 90])
x_test = x_test.reshape(1, 3, 1)
model.predict(x_test, verbose=0)

I can not understand whats happen with input data inside all 100 LSTM cells when I invoke model.predict().

Explain me please what all this 100 units do if x_test array goes into one unit input.

Answer 1

The test data is a 3 step sequence data i.e. 70,80,90.

Following will be the flow -

• 70 will reach the 1st LSTM layer i.e. to the all 100 in parallel
• It will also come back by the Recurrent weight to be multiplied with the next seq.
• 80 will reach and add up with the Recurrent of 70.
• Similarly, 90 will pass through the 1st LSTM layer.
• All the seq_step is held at 1st layer because return_sequences=Flase
• After the last seq_step, the output will reach the next layer
• RepeatVector will duplicate the vector, so the single output will become 2 step input for the next LSTM layer.
• When the first of the duplicate will reach the LSTM, it will behave exactly like the previous LSTM layer. In addition, it will pass the output to the next layer too because return_sequences=True here.
• Same thing will repeat for the 2nd duplicate vector.
• The role of TimeDistributedlayer is to wrap the two incoming data as two steps and hence you will receive a (2,) output.

How vectors look across the Recurrent Layers

• This is intuitive for a single Neuron/Single Feature i.e. as depicted below

$\hspace{5cm}$

• For a multi-feature, multiple Neuron, it goes like below
  Your case has a single feature for each step and 100 Neurons. The below depiction has 3 features and 4 Neurons. Features behave the way it does in a simple feedforward network. The only new information here is that recurrent of each Neuron goes back to each of the 100.
  
  $\hspace{7cm}$ ^{Image credit - SO Answer}