1
$\begingroup$

I am an beginner in text generation and and deep-learing but I like to get in touch with it. Currently I am learing about LSTM networks and VAEs for text generation. I would like to read a sequence at once and output another sequence. What I learned from this post is that the input shape should be 3D. Since my text data is 2D I want to reshape it. But this is now the confusing part. As I have training data with the shape of (nb_sequences, max_seq_lenth) = (274, 95). As the post pointed out, it would be important to reshape the data to a format of input = (nb_sequence, nb_timestep, nb_feature) with nb_sequence being the number of sequences (274), nb_timestep the length of the sequence (95) and nb_feature the number of features describing a sequence at a timestep, right?

I have two questions here:

1). How to reshape properly? My vocab size is 359 which I imagined (obviously wrong) to be the number of the features. --> input = (274, 95, 359). This is not possible since an array of 274*95 = 26030 can't be reshaped into (274, 95, 359).

If I take the number of diffrent words my sequences are containing, wouldn't it lead to some erros since sentences do not consist out of the same words naturally? Meaning that I would have a variable feature number with each sequence?

2). How would one realize the problem I am facing? I read about one-hot encoding the sequences. This might solve the problem I have regarding question 1), but as far as I understand this would lead me to a quite inefficient way of predicting sequences as I want to handle more data. Is there any suggestion how to do it in a effient way?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

The input to an LSTM must be a batch of sequences of vectors of real numbers, i.e. 3D tensor). Textual inputs are discrete tokens, so they are a batch of sequences of integers (indexes to the vocabulary table), i.e. a 2D tensor.

Before any reshaping, you must transform each integer value into a vector. For that, the usual approaches are one-hot encoding and embedding tables:

  • With one-hot encoding, you encode each integer index as a vector with 359 elements, where all are 0s except the position at the integer index, which is a 1.

  • With an embedding table, you keep a table with 359 vectors of an arbitrary dimensionality (the embedding size).

In either case, you get the "extra dimension" that you were lacking with integer indices.

Improve this answer
$\endgroup$
2
  • $\begingroup$ do you have a suggestion, where I can read about losses and metrics used for text generation? I also find this quite confusing because in the posts and papers I read, the most people don't tell why they are using a specific loss function $\endgroup$
    – Tim
    Commented May 21, 2021 at 6:42
  • $\begingroup$ The loss for text generation is almost always the categorical cross-entropy, which is appropriate for discrete token selection. About performance metrics, this depends on the actual task, and it is normally a "cultural thing". For instance, in machine translation, we use the BLEU score; we know it is flawed, but the alternatives are more complex and also have flaws. $\endgroup$
    – noe
    Commented May 21, 2021 at 7:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.