Have you guys tried to compare the performance of TF-IDF features* with a shallow neural network classifier vs a deep neural network models like an RNN that has an embedding layer with word embedding as weights next to the input layer? I tried this on a couple of tweet datasets and got surprising results: f1 score of~65% for the TF-IDF vs ~45% for the RNN. I tried the setup embedding layer + shallow fully connected layer vs TF-IDF + fully connected layer but got almost same result difference. Can you guys give some opinion on how TF-IDF features can outperform the embedding layer of a deep NN? Is this case common? Thanks!

  • I've used unigrams and bigrams to produce the TF-IDF features

It is common for TFIDF to be a strong model. People constantly get high places in Kaggle competitions with TFIDF models. Here is a link to the winning solution that used TFIDF as one of its features (1st place Otto product classification). You will most likely get a stronger model if you combine the TFIDF and RNN into one ensemble. Other results from Kaggle:


A good number of kernels are going the traditional route with CountVectorizer/TF-IDF, and some brave souls (I say brave because training is slower and the results don't seem as spectacular so far) have been experimenting with embeddings, as per the previous competitions.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.