how to use word embedding to do document classification etc?

Question

I just start learning NLP technology, such as GPT, Bert, XLnet, word2vec, Glove etc. I try my best to read papers and check source code. But I still cannot understand very well.

When we use word2vec or Glove to transfer a word into a vector, it is like:

[0.1,0.1,0.2...]

So, one document should be like:

[0.1,0.1,0.2...]
[0.1,0.05,0.1...]
[0.1,0.1,0.3...]
[0.1,0.15,0.1...]
.......

So, one document is a matrix. If I want to use some traditional method like random forest to classify documents, how to use such data? I was told that Bert or other NLP models can do this. But I am really curious about how the word embedding are applied in the traditional methods?

Answer 1

So, one document is a matrix. If I want to use some traditional method like random forest to classify documents, how to use such data?

You can't, at least not directly because traditional methods require a fixed number of features for every instance. In the case of document classification the instance must represent the document, so unless all the documents have exactly the same length (unrealistic) it's impossible to use a set of vectors as features.

The traditional approach would consist in representing a document with a vector where each cell represents a word in the vocabulary, and the value is for instance the TFIDF weight of the word in the document.

Answer 2

Word embedding techniques can be adapted to obtain a representation of each document as a single vector, i.e. "Doc2Vec". A naive approach to Doc2Vec is to simply sum the word embedding vectors in each document, and divide each element in this vector by its length. For a better approach to Doc2Vec, see Mikolov et al's paper: Distributed Representations of Sentences and Documents.

Once you obtain the Doc2Vec representations of each document, you can then apply traditional classification algorithms to this.

Alternatively, you can use the matrices of word embeddings directly as the first layer in a convolutional neural network: see here.