How should I format input and output for text generation with LSTMs

Question

I'm attempting to generate a response to an input line of text using an LSTM. I've considered various forms of input, including one-hot encoding each character in the line and passing each input line as a vector of one-hot encoded vectors. I've also considered using a dictionary and one-hot encoding each word in the sentence based on its alphabetical position.

However, I'm not sure about any of this, as I am new to natural language processing in machine learning. What would be the best way to format my input (and my output) for this problem?

Answer 1

You can use word embedding, to encode words as vectors of real numbers. Then all calculations, such as comparison of words (to find similarity), are performed in that high-dimensional space.

"What would be the best way to format my input (and my output) for this problem?"

I cannot tell which is the best approach (depends on your problem) but this one is very common. It's also very easy to implement in Keras, using the Embedding layer.

Answer 2

Ultimately, the exact representation of your input will be dependent on the tool you are feeding it in.

More generally, for text generation, you will want to model your input and output as sequences of tokens. These tokens can be words, sentences, characters, n-grams, whatever floats your boat.

Each token should then be represented by a vector. That vector could be a one-hot encoding of the token, or as pcko1 suggests, a word embedding. Word embeddings are real vectors which represent your token. They can be used in the same way as one-hot vectors, but they have shown to carry more meaning.

Ultimately, you will need to have some sort of vector_to_string(vector) and string_to_vector(string) functions, which will map a token to its corresponding vector, and vice versa. That way, you transform your input string into a sequence of vectors, and then, your output which will be a sequence of vectors can be turned back into a string.

For text generation, it is useful to add a start_of_sentence and end_of_sentence tokens, to know when to stop generating.