I am trying to build a multi-label classifier for suggesting tags on blog posts. The textual data is full of noise. The approach I have been following until now was a BOW approach with Tf-idf weighting. However I could only get an accuracy of around 0.35 on the test set using OnevsRest approach and a SVC. I then decided to eliminate the noise and I applied TextRank on my data to obtain a more meaningful summary and then applied the same BOW approach which however resulted in further loss of accuracy. I am quite new to machine learning and data science in general. Can someone please guide me with the approach? I am looking specifically on how to create my features and what else can I try. I have 12 labels and around 1500 blog posts. I know it isn't much data, but the data-set is still well-balanced and won't be the primary reason for such low accuracy.
-
$\begingroup$ Are you normalising the tf-idf vectors? Removing stop words? Cleaning the text? What is the distribution of the data (i.e. class labels)? $\endgroup$– HarpalJul 22, 2016 at 16:16
-
$\begingroup$ Hadn't normalized. It didn't change much. Yes, I had removed all the stop words and cleaned the text, lemmatized, stemmed, almost all the basic processes. $\endgroup$– Himanshu RaiJul 26, 2016 at 11:28
3 Answers
What you seem to want is to identify meaningful discrepancies between your blog posts in order to identify what category they fit in.
I think it's important for you to consider the classification problem in terms of human thinking:
- How would humans decide which blog gets which tags?
- Can we distil this decision-making into meaningful features? In other words, does this decision-making depend on (linguistic) phenomena that can be found in the text? (it probably does)
- (How) can we extract these features from the text?
Also, it might be the case that either the tag set you propose or the data set you have are too similar. This means that the tags might be too similar and that their occurrence depends on many similar feature values which makes it difficult for your training algorithm to extract discriminative features. The same thing happens when the data expresses too little variety.
On a final note: This might well be the first attempt to solving this particular problem (on this data set and this set of labels), so the number you get is not necessarily bad or sub-par, because it might simply be a difficult task.
-
$\begingroup$ Thanks. I wanted to share one more thing, while using LinearSVC I looked at the predictions on my test set and what I observed was that all entries turned out to be classified in one or none of the labels. Any idea why this might be happening? $\endgroup$ Jul 21, 2016 at 5:22
-
$\begingroup$ @HimanshuRai As far as I know, LinearSVC needs a little bit of tweaking in order to allow for multiclass classification, whereas SVC can do it out of the box. Read this: scikit-learn.org/stable/modules/… $\endgroup$– lennyklbJul 21, 2016 at 7:35
-
$\begingroup$ Yeah, I wrapped it in a OneVsRest Classifier. $\endgroup$ Jul 21, 2016 at 10:26
-
$\begingroup$ Can you tell me in what scenario might an SVM predict no classes? $\endgroup$ Jul 21, 2016 at 10:38
BOW approach with Tf-idf weighting seems like a good strategy however you need to improve your feature engineering strategy. I would recommend to include n-grams (bigrams, trigrams etc.), special charecters (exclamations), emoji unicodes etc.
0.35 is a poor estimate than of a random accuracy (<=0.5). I think your model is definitely overfitting. I think your dataset may have class imbalance and may need to account for that.
I know why your LinearSVC is giving 0 or 1 class results only: when you are using one vs rest, you have unbalanced dataset. You must add class_weight=balanced to your estimator, because rest set is much bigger.