3
$\begingroup$

I am trying to implement an LSTM structure in plain numpy for didactic reason. I clearly understand how to input the data, but not how to output. Suppose I give as inputs a tensor of dimension (n, b, d) where: • n is the length of the sequence • b is the batch size (timestamps in my case) • d the number of features for each example Each example (row) in the dataset is labelled 0-1. However, when I fed the data to the LSTM, I obtain as a result the hidden state h_out which has the same dimension of the hidden size of the network. How can I obtain just a number that can be compared to my labels and properly backpropagated? I read that someone implements another dense layer on top of the LSTM, but it's not clear to me the dimensions that such layer and its weight matrix should have.

$\endgroup$
2
$\begingroup$

What you are getting as the output is the internal LSTM state. In order to get value comparable to your labels, add a dense layer on top of it. Output dimension of dense layer would be the number of labels you want result.

  1. If its 0 and 1, only 1 output neuron can work along with sigmoid
  2. If there are 5 label classes, then output dimension of dense layer should also be 5
$\endgroup$
  • $\begingroup$ That's clear. I have some more questions about the dense layer. If my output of the hidden state of LSTM is of dimension (timesteps,hidden_size) for each sequence processed, should the weight matrix of the dense layer be of dimension (hidden_size,1)? In this way I should obtain a vector made of (timesteps,1) elements, which is the output I want because I want to compare it to labels at each timestep. I hope this is the reasoning and the dimensionality that the layer should have. Furthermore, since my labels are 0/1, which activation should I use for such layer? $\endgroup$ – Alexbrini Dec 31 '18 at 9:12
  • 1
    $\begingroup$ For 0/1 use sigmoid activation function $\endgroup$ – user5722540 Dec 31 '18 at 9:16
  • $\begingroup$ Ok. Is the reasoning behind the dimensionality of the dense layer correct? It's my first ML application and I'm not sure it's the correct way to obtain classification label at each timesteps, but it seems correct to me in that way. $\endgroup$ – Alexbrini Dec 31 '18 at 9:19
  • $\begingroup$ It's correct. I would recommend give it a shot. Best stuff is learnt through errors. Post any error you face upon Implementation. $\endgroup$ – user5722540 Dec 31 '18 at 9:22
  • $\begingroup$ I'm stuck with the backward propagation part. Suppose I use the Euclidean norm as a cost function. I start propagating back the error through the fully connected layer. Then I need to consider the activation of the previous layer in order to update the weights of the dense layer. Is the activation of the previous function exactly the hidden state resulting from the LSTM layer? I am trying but the dimensions don't match and I haven't found anything about the implementation from scratch of similar architectures. $\endgroup$ – Alexbrini Jan 2 at 16:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.