0
$\begingroup$

I learn the LSTM recently, and a little bit confuse about the model parameters about LSTM,

The follow is the LSTM structure

enter image description here

And it is equation as (I slightly ignore the bias in each equation):

$$i_t = \sigma(x_tU^i + h_{t-1}W^i)$$ $$f_t = \sigma(x_tU^f + h_{t-1}W^f)$$ $$o_t = \sigma(x_tU^o + h_{t-1}W^o)$$ $$\tilde{C}_t = \tanh(x_tU^g + h_{t-1}W^g)$$ $$C_t=\sigma(f_t*C_{t-1} + i_{t}*\tilde{C}_t)$$ $$h_t = \tanh(C_t)*o_t$$

I am confused about the parameters: $U^i, U^f, U^o , U^g$ and $W^i, W^f, W^o , W^g$. Are those parameters trained by data(e.g., back-propagation)?

My second question is that, In the Figure, We have the three block of LSTM 1, 2, and 3 (mark by blue color). Do them share the same $U^i, U^f, U^o , U^g$ and $W^i, W^f, W^o , W^g$. ? Or each block have their own $U^*$ and $W^*$?

$\endgroup$
0
$\begingroup$

I am confused about the parameters: $U^i, U^f, U^o , U^g$ and $W^i, W^f, W^o , W^g$. Are those parameters trained by data(e.g., back-propagation)?

Yes. They do get trained by backpropagation. There are caveats here. Variants to backprop too exist when training a sequence-to-sequence to models. Check out BPTT for example.

My second question is that, In the Figure, We have the three block of LSTM 1, 2, and 3 (mark by blue color). Do them share the same $U^i, U^f, U^o , U^g$ and $W^i, W^f, W^o , W^g$. ? Or each block have their own $U^*$ and $W^*$?

Each block has the same set of weights shared among them.

An following mini example demo would be like in Keras.

from keras.models import Sequential
from keras.layers import Dense, Dropout
from keras.layers import Embedding
from keras.layers import LSTM
model = Sequential()
model.add(LSTM(128, input_shape=(None, 100)))
model.add(Dropout(0.5))
model.add(Dense(1, activation='sigmoid'))
model.compile(loss='binary_crossentropy',
              optimizer='rmsprop',
              metrics=['accuracy'])

creates an simple LSTM model. If you look at model summary to count the params you got.

model.summary

You will get

_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
lstm_3 (LSTM)                (None, 128)               117248    
_________________________________________________________________
dropout_2 (Dropout)          (None, 128)               0         
_________________________________________________________________
dense_2 (Dense)              (None, 1)                 129       
=================================================================
Total params: 117,377
Trainable params: 117,377
Non-trainable params: 0
_________________________________________________________________

And the total of 117,377 params you see is obtained from (((128*128)+(128*100)) * 4) + (4*128). Here W is of size 128*100 and U is 128*128 and there are 4 of them resp. and last 4*128 goes for the biases.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks. Correct me if I am wrong, (1) In the forward, $W$ and $U$ are the same for cell 1,2 and 3. (2) in the "Backpropagation through time", I don't quite understand. Does $W$ and $U$ update for all three cell at the same time? Or we first update $W, U$ in cell 3, then, update $W, U$ in cell 2, then in cell 1. If we do this, the next forward, the $W, U$ now have different value for cell 1,2 and 3. $\endgroup$ – jason Mar 3 '19 at 15:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.