Models after word2vec outputs

Question

I am originally using a bag of word (2-gram) model to approach a classification problem. The one hot encoding of the 2-gram output was sent to a logistic regression or neural network to build a classification model.

Now, I am experimenting the gensim word2vec approach, each word is now a vector from word2vec. That is, if my sentence have 10 words, it would become a 10x30 array (assume word2vec embedding dimension is 30). It's not clear to me how do I send such outputs to a logistic regression or neural network model like before ...

What type of model should I use after the gensim word2vec outputs to solve a classification problem? Thanks!

Answer 1

There exists multiple ways, each consisting on summarizing the embeddings of each word from a document.

The most common ones when using linear models like Logistic Regression are:

• max of word vectors
• mean of word vectors
• concatenation of word vectors (but implies a fixed length)

However, it is important that using such approaches often results in :

• either equal or worse performance than using bag-of-words
• less interpretability (if applicable)

The reason is the loss of information (especially for long documents).

When using word embeddings (and so called word2vec), the best choice is often to use models that can handle a second dimension in your input.

For that, recurrent and convolutional neural networks have proven to be effective. The former operates at the word level by recurrently feeding the word embeddings, the latter operates at the whole document level (represented by a matrix of $n_{words}\times d_{embedding}$ as you mentioned) by locally convoluting filters with windows of the document.