3
$\begingroup$

I'm new to Recurrent Neural Network but I want to train my data with LSTM but I'm having a trouble to understand LSTM Many-to-One architecture. Suppose the size of my data is time_step x num_features say 2 x 2 and I have to use many-to-one LSTM architecture because I want to do classification. So in the last time_step I have to add dense (a) which contain sigmoid activation function to predict the sequence class which is 0 or 1.

My questions are,

  1. When I compute a, do I need to include all the hidden state (h1 and h2) or just the last hidden state h2?
  2. If I just include the h2, how do I calculate the derivatives of the loss function (cross entropy) w.r.t h1?

The derivation for h1 is highly recommended. Thank you :)

Improve this question
$\endgroup$
  • $\begingroup$ By "compute", do you mean derivation of weights / biases during back prop ? $\endgroup$ – Shamit Verma Jan 29 '19 at 16:31
  • $\begingroup$ Yes, because I got confused when computed the sigmoid activation function in the dense layer. $\endgroup$ – Mei Lie Jan 29 '19 at 16:45
0
$\begingroup$

While training, a set of training examples will be provided in a batch. At end of each batch, weights for all layers are updated (Dense and LSTM).

https://adventuresinmachinelearning.com/keras-lstm-tutorial/

Improve this answer
$\endgroup$
  • 1
    $\begingroup$ I know that but it doesn't answer my question. What I'm confused is when calculate the sigmoid function which is a=sigmoid(w.T h + b) where h is the hidden state. I need to know what are hidden state which is being calculate inside sigmoid function. Whether the last hidden state (at the last time step) or all the hidden state (from the first time step to the last) $\endgroup$ – Mei Lie Jan 29 '19 at 17:01
  • $\begingroup$ Sigmoid has only 1 state. Update for tat state is calculated only once in a batch. $\endgroup$ – Shamit Verma Jan 30 '19 at 4:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.