I am trying to implement an LSTM structure in plain numpy for didactic reason. I clearly understand how to input the data, but not how to output. Suppose I give as inputs a tensor of dimension (n, b, d) where: • n is the length of the sequence • b is the batch size (timestamps in my case) • d the number of features for each example Each example (row) in the dataset is labelled 0-1. However, when I fed the data to the LSTM, I obtain as a result the hidden state h_out which has the same dimension of the hidden size of the network. How can I obtain just a number that can be compared to my labels and properly backpropagated? I read that someone implements another dense layer on top of the LSTM, but it's not clear to me the dimensions that such layer and its weight matrix should have.
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
5
What you are getting as the output is the internal LSTM state. In order to get value comparable to your labels, add a dense layer on top of it. Output dimension of dense layer would be the number of labels you want result.
- If its 0 and 1, only 1 output neuron can work along with sigmoid
- If there are 5 label classes, then output dimension of dense layer should also be 5
-
$\begingroup$ That's clear. I have some more questions about the dense layer. If my output of the hidden state of LSTM is of dimension (timesteps,hidden_size) for each sequence processed, should the weight matrix of the dense layer be of dimension (hidden_size,1)? In this way I should obtain a vector made of (timesteps,1) elements, which is the output I want because I want to compare it to labels at each timestep. I hope this is the reasoning and the dimensionality that the layer should have. Furthermore, since my labels are 0/1, which activation should I use for such layer? $\endgroup$ Commented Dec 31, 2018 at 9:12
-
1$\begingroup$ For 0/1 use sigmoid activation function $\endgroup$ Commented Dec 31, 2018 at 9:16
-
$\begingroup$ Ok. Is the reasoning behind the dimensionality of the dense layer correct? It's my first ML application and I'm not sure it's the correct way to obtain classification label at each timesteps, but it seems correct to me in that way. $\endgroup$ Commented Dec 31, 2018 at 9:19
-
$\begingroup$ It's correct. I would recommend give it a shot. Best stuff is learnt through errors. Post any error you face upon Implementation. $\endgroup$ Commented Dec 31, 2018 at 9:22
-
$\begingroup$ I'm stuck with the backward propagation part. Suppose I use the Euclidean norm as a cost function. I start propagating back the error through the fully connected layer. Then I need to consider the activation of the previous layer in order to update the weights of the dense layer. Is the activation of the previous function exactly the hidden state resulting from the LSTM layer? I am trying but the dimensions don't match and I haven't found anything about the implementation from scratch of similar architectures. $\endgroup$ Commented Jan 2, 2019 at 16:35