I am trying to build a multi-label classifier for suggesting tags on blog posts. The textual data is full of noise. The approach I have been following until now was a BOW approach with Tf-idf weighting. However I could only get an accuracy of around 0.35 on the test set using OnevsRest approach and a SVC. I then decided to eliminate the noise and I applied TextRank on my data to obtain a more meaningful summary and then applied the same BOW approach which however resulted in further loss of accuracy. I am quite new to machine learning and data science in general. Can someone please guide me with the approach? I am looking specifically on how to create my features and what else can I try. I have 12 labels and around 1500 blog posts. I know it isn't much data, but the data-set is still well-balanced and won't be the primary reason for such low accuracy.
What you seem to want is to identify meaningful discrepancies between your blog posts in order to identify what category they fit in.
I think it's important for you to consider the classification problem in terms of human thinking:
- How would humans decide which blog gets which tags?
- Can we distil this decision-making into meaningful features? In other words, does this decision-making depend on (linguistic) phenomena that can be found in the text? (it probably does)
- (How) can we extract these features from the text?
Also, it might be the case that either the tag set you propose or the data set you have are too similar. This means that the tags might be too similar and that their occurrence depends on many similar feature values which makes it difficult for your training algorithm to extract discriminative features. The same thing happens when the data expresses too little variety.
On a final note: This might well be the first attempt to solving this particular problem (on this data set and this set of labels), so the number you get is not necessarily bad or sub-par, because it might simply be a difficult task.
BOW approach with Tf-idf weighting seems like a good strategy however you need to improve your feature engineering strategy. I would recommend to include n-grams (bigrams, trigrams etc.), special charecters (exclamations), emoji unicodes etc.
0.35 is a poor estimate than of a random accuracy (<=0.5). I think your model is definitely overfitting. I think your dataset may have class imbalance and may need to account for that.