Usage of Word2Vec

Question

Sorry for the basic doubt,

I would like to know if I can use my Word2Vec straight for classification without using LSTM. My assumption is it’s not possible because the ordering of the words will not be take into account. Hence it wont perform for classification.

But we use BERT embeddings for classification. But in this case, BERT generates embedding based on context of the sentence. Hence we can use it for classification. Is my understanding right?

BERT achieves what LSTM learns through ordering without sequential processing, It finds an embedding by processing the sentence as a whole. LSTM also tries to represent a sentence using some context but does it through sequential processing. Is my understanding right?

Answer 1

Sure, you can average the word2vec vectors of all words in the sentence and train a linear classifier with labeled data. Before doing that, you may remove stopwords that do not add meaning.

This approach, of course, disregards word order, so your results may not be good depending on the kind of text you are classifying.

If you google "word2vec text classifications" you will find many helpful resources.

Answer 2

Word2Vec word vectors were being used for classification purposes in the early days. We could take word vectors of all the words in the sentence to get the resultant vector which could give us some sense of what this sentence would mean but this was not enough for more complex scenarios where we need to extract more information from the sentence. for e.g. what this sentence is referring to etc.

Then came LSTM which tries to learn the information by using the relative ordering of the words in the sentence. It tries to learn what to hold after processing a few words(limitation of the architecture) in the order and using that information along with the next word vector to output the next information to hold. So it is always the interaction of a word with the overall information carried up to that time and the actual info for each word gets diluted as more words are processed.

To deal with this issue, BERT calculates dot product(simplifying a bit) for each word with the other words in the sentence(aka attention matrix) thus capturing every information it can. The dot product for each pair of words does not necessarily make sense but more or less they represent how much they 'attend' to each other. While training we usually fine-tune the attention matrix to give the right attention values in order to have correct predictions. This idea is not enough because it again misses the relative ordering information for the words. So they added positional encodings to each word based on where it lies in the sentence. for eg. 1st occurrence of 'to' will have some x positional encoding added while 2nd occurrence of 'to' will have y position encoding added to its vector, both will have same vector in the starting.

This idea also made better predictions for classification tasks as well as for question answering and other complicated tasks which were not possible before BERT.