0
$\begingroup$

I'm developing an LSTM neural network algorithm that, for lack of a better summary, takes a question as input and generates an answer as output. Now, the way I'm going about it is to parse the question word-by-word and have the algorithm note some of the previous words to implicitly learn how these words combine and affect the answer. For example, the question "How's the weather in Philadelphia?" gets broken up into "How's","the", "weather", "in","Philadelphia".

The problem is, not all my questions or answers are of the same length. I could have questions like, "How's the weather?" or "What color of pants am I wearing?" Plus, the vocabulary consists of about 300 words, making a one-hot encoding scheme based simply on these words impractical.

$\endgroup$

1 Answer 1

1
$\begingroup$

Typically, people do not use one hot encoding as it takes up a lot of space (vocabulary size explodes) and is difficult to learn as well. Instead, they project words into a high dimensional vector space where each dimension represents some sense or direction.

For example, if you consider the words red and green - they are opposites in the context of a traffic light but are similar in the sense that they are both names of colours. A vector representation of the word will incorporate such relations by having similar values for some dimensions and different values for the others.

For more details, you can read this paper.

As for your next question, standard practice is to limit the sentence length through truncation and pad smaller sentences with additional 0 vectors.

$\endgroup$
2
  • $\begingroup$ Thanks! I have just one question. How does vectorizing my data help? For example, for the skip-gram model, I need to have a vector for each possible combination of words ("The","quick"),("The","brown"),("The","fox") etc. How does this help me compress my data? $\endgroup$
    – moonman239
    Commented Jul 26, 2018 at 19:24
  • $\begingroup$ I don't know why you think you have to make a vector for each combination of two words. Can you explain what you are saying via a use case? In the skip gram model, you will have |V| vectors, each of size dim which may be something like 50-200. $\endgroup$
    – doodhwala
    Commented Jul 26, 2018 at 20:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.