This is in continuation of my earlier post.
In my previous model, I used just two features which couldn't fare well and gave 71% prediction (accuracy) score. Now, I'm trying to consider another important field "subject" from the data set. I'm planning to use NLTK package along with Tf-Idf.
I generated the below using NLTK (FreqDist) on the subject field - Here is my idea: I'll tokenize the subject field, apply stopwords, then Tf-Idf and get the top N most frequent words and use them as features. The Tf-Idf score will be my values. In total, I'll have (N+2) features.
But, one downside is - the feature matrix will be huge (say 200+ features) and sparse. Is that a concern? Does this design make sense?
Your thoughts please.