In a huge dataset for NLP it is taking very long time to classify my dataset

therefore, trying each feature extraction method separetly is time consuming and not effecient.

Is there a way that can tell me which method (TFIDF or Bag Of Words) is more likely to give the highest F1 score.

I tried test them on smaller subset (1000 records) it was fast but best method in smaller subset does not mean it is the best in complete dataset.

any other way to decide which method to use?


There is no specific way to deal with these kinds of experimentation. Below are some important points to remember before doing experimentation

  1. If you are using NN to do the work, dense vectors like word2vec or fasttext may give better results than BoW/TfIdf

  2. If you have more OOV words then fasttext may give better output than basic Word2Vec

  3. If you are using linear algorithms like Logistic Regression/Linear SVM, BoW/TfIdf may have some advantage over averaging all the word vectors in the sentence. But it's not always true.

  4. For the tree-based algorithms, training time may increase if we use BoW/TfIdf features because of the huge feature-length.

  • $\begingroup$ What is OOV? ?? $\endgroup$ – asmgx Mar 3 at 8:06
  • $\begingroup$ Out of Vocab words $\endgroup$ – Uday Mar 3 at 8:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.