How to select a proper vectorization method in NLP?

Question

Suppose we have a text classification problem.

As we all know we have to convert the text data into vectors so as to train the model. So there are a couple of vectorization methods such as count vectorization, Tf-IDF, Bag of words, etc. So from these many vectorization methods how will we choose one method? Is it like that or in another way do we need to try all the methods, train the model then check the performance with each vectorization method?

Please share your thoughts and help me to understand this properly.

Answer 1

Count Vectorizer is a way to convert a given set of strings into a frequency representation.Count Vectors can be helpful in understanding the type of text by the frequency of words in it. But its major disadvantages are:

Its inability in identifying more important and less important words for analysis.
It will just consider words that are abundant in a corpus as the most statistically significant word.
It also doesn't identify the relationships between words such as linguistic similarity between words.

TF-IDF is better than Count Vectorizers because it not only focuses on the frequency of words present in the corpus but also provides the importance of the words. We can then remove the words that are less important for analysis, hence making the model building less complex by reducing the input dimensions.

Even though TFIDF can provide a good understanding about the importance of words but just like Count Vectors, its disadvantage is:

It fails to provide linguistic information about the words such as the real meaning of the words, similarity with other words etc.

So, you should try to align your use case with pros mentioned above. Also i would suggest to explore Word2vec and Glove Embeddings if you wants vectors to have some contextual information