This is in continuation of my earlier post.

In my previous model, I used just two features which couldn't fare well and gave 71% prediction (accuracy) score. Now, I'm trying to consider another important field "subject" from the data set. I'm planning to use NLTK package along with Tf-Idf.

I generated the below using NLTK (FreqDist) on the subject field - enter image description here Here is my idea: I'll tokenize the subject field, apply stopwords, then Tf-Idf and get the top N most frequent words and use them as features. The Tf-Idf score will be my values. In total, I'll have (N+2) features.

But, one downside is - the feature matrix will be huge (say 200+ features) and sparse. Is that a concern? Does this design make sense?

Your thoughts please.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.