# Understanding LSTM text input

I am an beginner in text generation and and deep-learing but I like to get in touch with it. Currently I am learing about LSTM networks and VAEs for text generation. I would like to read a sequence at once and output another sequence. What I learned from this post is that the input shape should be 3D. Since my text data is 2D I want to reshape it. But this is now the confusing part. As I have training data with the shape of (nb_sequences, max_seq_lenth) = (274, 95). As the post pointed out, it would be important to reshape the data to a format of input = (nb_sequence, nb_timestep, nb_feature) with nb_sequence being the number of sequences (274), nb_timestep the length of the sequence (95) and nb_feature the number of features describing a sequence at a timestep, right?

I have two questions here:

1). How to reshape properly? My vocab size is 359 which I imagined (obviously wrong) to be the number of the features. --> input = (274, 95, 359). This is not possible since an array of 274*95 = 26030 can't be reshaped into (274, 95, 359).

If I take the number of diffrent words my sequences are containing, wouldn't it lead to some erros since sentences do not consist out of the same words naturally? Meaning that I would have a variable feature number with each sequence?

2). How would one realize the problem I am facing? I read about one-hot encoding the sequences. This might solve the problem I have regarding question 1), but as far as I understand this would lead me to a quite inefficient way of predicting sequences as I want to handle more data. Is there any suggestion how to do it in a effient way?

The input to an LSTM must be a batch of sequences of vectors of real numbers, i.e. 3D tensor). Textual inputs are discrete tokens, so they are a batch of sequences of integers (indexes to the vocabulary table), i.e. a 2D tensor.

Before any reshaping, you must transform each integer value into a vector. For that, the usual approaches are one-hot encoding and embedding tables:

• With one-hot encoding, you encode each integer index as a vector with 359 elements, where all are 0s except the position at the integer index, which is a 1.

• With an embedding table, you keep a table with 359 vectors of an arbitrary dimensionality (the embedding size).

In either case, you get the "extra dimension" that you were lacking with integer indices.

• do you have a suggestion, where I can read about losses and metrics used for text generation? I also find this quite confusing because in the posts and papers I read, the most people don't tell why they are using a specific loss function
– Tim
May 21, 2021 at 6:42
• The loss for text generation is almost always the categorical cross-entropy, which is appropriate for discrete token selection. About performance metrics, this depends on the actual task, and it is normally a "cultural thing". For instance, in machine translation, we use the BLEU score; we know it is flawed, but the alternatives are more complex and also have flaws.
– noe
May 21, 2021 at 7:16