How to overcome training example's different lengths when working with Word Embeddings (word2vec)

Question

I'm working on Sentiment Analysis over tweets using word2vec as word representation.

I have trained my word2vec model. But when I'm going to train my classifier, I'm facing the issue that every tweet has different length and the classifier (RandomForest) needs all examples to be of the same size.

Currently, for every tweet I'm averaging the vectors of all of its words, to end up with a vector representation of the tweet.

For example: My word2vec model represents each word as vectors of size 300.

I have Tweet1 formed by 10 words, and Tweet2 formed by 5 words.

So what I do is, for Tweet1

(v1_Tweet1 + v2_Tweet1 + ... +v10_Tweet1)/10 = v_Tweet1 #avg vector of 300 elements.

For Tweet2:

(v1_Tweet2 + v2_Tweet2 + ... +v5_Tweet1)/5 = v_Tweet2 #avg vector of 300 elements.

*Being v1_TweetX the vector of the first word of the TweetX and so on.

This works 'fine' but I would like to know what other approaches do you take to overcome the different sizes on the train and text examples for the classifier.

Thanks.

Answer 1

Let me suggest three simple options:

1. average the vectors (component-wise), i.e., compute the word embedding vector for each word in the text, and average them. (as suggested by others).

2. take the (component-wise) maximum of the vectors. (max, instead of average)

3. take the (component-wise) minimum of the vectors. (min, instead of average)

Each of these yields a feature vector that is independent of the length of the text.

There is some research suggesting that concatenating the max and the min yields a pretty effective feature space: it's not the absolute optimal, but it's close to optimal, and is simple and easy to implement. See this question on Statistics.SE for details.

---

Here is an alternative idea, inspired by cubone's answer, that as far as I know hasn't been tested before. The idea is to tag the text using a part-of-speech tagger and then use those tags to inform the featurization process.

In particular, write down a list of all possible POS tags that could be emitted by the POS tagger. Suppose there are 20 possible tags (CC, DT, JJS, MD, NNP, ...). Then the feature vector will be 20*300 = 6000 elements long: it will have one 300-vector per POS tag, concatenated in some canonical order. The 300-vector for each tag could be computed by averaging the word embedding vectors of all words that are tagged by the POS-tagger with that tag. Or, you could get one 600-vector per POS tag, obtained by computing the min and max over all vectors of words with that tag.

This might yield a richer feature space, I don't know if it would yield any improvement, but it's something you could try if you wanted to experiment with different ideas.

Answer 2

Two very different suggestions here to avoid averging the vectors:

1. Use Word Mover's Distance (https://github.com/mkusner/wmd) to compute distance between the tweets (not sure how well it would work on short texts like tweets, I still need to try that myself...)
2. Cluster the word vectors themselves (using e.g. kmeans), then for each tweet create a vector with k entries (one for each cluster) that encodes whether it contains words belonging to that cluster. I think I saw this in a Kaggle tutorial on word2vec, will be happy for the link if you find it!

Answer 3

Instead of averaging and getting a single vector for the tweet, you can instead get vectors for each word and for different length vector sizes, padding can be done with zeros.

Answer 4

In my work, I have done the same way by averaging the word vectors. But there is another idea I wanted to try. It is with the help of POS tags.

First construct a most complicated sentence with all the POS tags as possible and set these POS tags as a template. For each sentence in the twitter corpus, POS tag all the words in it and apply those word vectors respective to the POS tags in the template. So, the unseen POS tags will have zeros.

For example: NNP PDT DT NNS VB MD JJS CC PRP RBS is the template. Thus each position will contain 300-dimensional vector totally a 3000-dimensional vector. And if the first tweet's POS tags are NNP VB JJS PRP, then the word vectors are applied on these positions and have vectors on NNP VB JJS PRP positions and 300-dimensional zero vectors on other positions.

This method not only solves the problem of representing the sentence by a single vector but also preserves the syntactic structure of the sentence by positioning in the right POS.

Ofcourse, there will be problems when there are more than one POS tags or jumbled positions of the tags. This is just one of the possibility.

Answer 5

I can think of a few possibilities that might fit your usecase:

1. Use the hasshing trick to turn arbitrary length vectors into constant length.
2. Trade out word vectors for paragraph vectors. Gensim and Deeplearning4j both have implementations you can look at.
3. Check out the awesome-2vec list which links to a tweet2vec implementation!