0
$\begingroup$

Suppose we have a text classification problem.

As we all know we have to convert the text data into vectors so as to train the model. So there are a couple of vectorization methods such as count vectorization, Tf-IDF, Bag of words, etc. So from these many vectorization methods how will we choose one method? Is it like that or in another way do we need to try all the methods, train the model then check the performance with each vectorization method?

Please share your thoughts and help me to understand this properly.

$\endgroup$

1 Answer 1

1
$\begingroup$

Count Vectorizer is a way to convert a given set of strings into a frequency representation.Count Vectors can be helpful in understanding the type of text by the frequency of words in it. But its major disadvantages are:

Its inability in identifying more important and less important words for analysis.
It will just consider words that are abundant in a corpus as the most statistically significant word.
It also doesn't identify the relationships between words such as linguistic similarity between words.

TF-IDF is better than Count Vectorizers because it not only focuses on the frequency of words present in the corpus but also provides the importance of the words. We can then remove the words that are less important for analysis, hence making the model building less complex by reducing the input dimensions.

Even though TFIDF can provide a good understanding about the importance of words but just like Count Vectors, its disadvantage is:

It fails to provide linguistic information about the words such as the real meaning of the words, similarity with other words etc.

So, you should try to align your use case with pros mentioned above. Also i would suggest to explore Word2vec and Glove Embeddings if you wants vectors to have some contextual information

$\endgroup$
2
  • $\begingroup$ As you mentioned, if our priority is for context we should consider word2vec and glove embeddings right? $\endgroup$
    – SRJ577
    Commented Jan 30, 2022 at 10:25
  • $\begingroup$ Absolutely correct. Tf-idf ,bag of word and other frequency based vectoriser cant take care of context. Please accept the answer if it was useful $\endgroup$ Commented Jan 30, 2022 at 13:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.