How do I store a question-and-answer combination for machine learning?

Question

I'm developing an LSTM neural network algorithm that, for lack of a better summary, takes a question as input and generates an answer as output. Now, the way I'm going about it is to parse the question word-by-word and have the algorithm note some of the previous words to implicitly learn how these words combine and affect the answer. For example, the question "How's the weather in Philadelphia?" gets broken up into "How's","the", "weather", "in","Philadelphia".

The problem is, not all my questions or answers are of the same length. I could have questions like, "How's the weather?" or "What color of pants am I wearing?" Plus, the vocabulary consists of about 300 words, making a one-hot encoding scheme based simply on these words impractical.

Answer 1

Typically, people do not use one hot encoding as it takes up a lot of space (vocabulary size explodes) and is difficult to learn as well. Instead, they project words into a high dimensional vector space where each dimension represents some sense or direction.

For example, if you consider the words red and green - they are opposites in the context of a traffic light but are similar in the sense that they are both names of colours. A vector representation of the word will incorporate such relations by having similar values for some dimensions and different values for the others.

For more details, you can read this paper.

As for your next question, standard practice is to limit the sentence length through truncation and pad smaller sentences with additional 0 vectors.