0
$\begingroup$

Hello I am trying to understand LSTMs but have a few problems:

  1. What is the input? Since LSTM is seq2seq I would think it is a sequence of words, but in a Codecademy lesson is mentioned that each sentence is represented as a matrix with a bunch of vectors containing 1 or 0 for the timestep -> sentence "I like Bobo" like = [0, 1, 0], so what is now the input? The matrix or the sequence of words?

  2. What is passed to the next LSTM cell after a prediction before was false? Since the false prediction is noted in the hidden state, how does the network know whether previous predictions were false? Or does it even know when predicting the next step?

I am excited for the answers, love Phiona.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

The input of an LSTM is a sequence of vectors. In your case, each of these vectors represents a word encoded as a one-hot vector. One-hot encoding is a way to express a discrete element (e.g. a word) numerically. Each one-hot vector is a vector of length $d$, where $d$ is the total number of words we can represent, and where all positions in the vector are 0 except the position associated with the represented word, which contains a 1.

The hidden state passed to the next LSTM cell is not the final binary prediction, but the dense numerical vectors we obtain before computing the binary prediction.

Improve this answer
$\endgroup$
7
  • $\begingroup$ Thanks! But: 1. So is it true that the input is a matrix containing vectors (sentences) containing vectors (words), where d (curvy) is the length of the (correct me when I am wrong) sentence with most words? I mean is the length for all vectors representing words in different sentences the same? $\endgroup$
    – Tknoobs
    Mar 1, 2021 at 18:26
  • 1
    $\begingroup$ With one-hot encoding, the dimensions of the input are: number of sentences $\times$ length of the longest sentence $\times$ number of representable words. $d$ is the number of representable words. You need to define $d$ a priori, that is, you have to select the vocabulary you are going to support, for instance, taking the $d$ most frequent words in your training data. $\endgroup$
    – noe
    Mar 1, 2021 at 18:42
  • $\begingroup$ Ok then d does not depend on the number of words in a sentence, but one the number of unique words over the whole dataset? I thought "dog" in the sentence "I love my dog" would have the vector [0, 0, 0, 1], because the words position is the 4. This is wrong, right? $\endgroup$
    – Tknoobs
    Mar 1, 2021 at 21:13
  • 1
    $\begingroup$ That's it, $d$ depends on the number of unique words in the whole dataset. And the vector is not because the word's position is the 4th. $\endgroup$
    – noe
    Mar 1, 2021 at 22:00
  • 1
    $\begingroup$ In classification, you normally wouldn't use an encoder-decoder architecture; these are normally used for sequence to sequence tasks, like machine translation. Normally, in encoder-decoder architectures, you have one LSTM for the encoder and a different one for the decoder. The decoder receives as input the expected output shifted one position to the right. As the comments are no place to discuss the details of that, I suggest you create a new question. $\endgroup$
    – noe
    Mar 2, 2021 at 13:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.