2
$\begingroup$

I'm doing some classification with a text corpus [professional emails], I've already done all the basic pre processing steps (stemming, remove stop words, punctuation, low frequency words, words length ...) which gives me a F1-score = 0.6.

I was wondering what could be the next step(s) to improve my results, I thought of "stacking" (use multiple classifiers) and maybe removing words which are common to the different classes (4 differents types of texts) [ not sure about this solution ]

Edit:

I tried with 15 differents classifier the best one is gradient boosting (gbm )with an F1-score=0.6002159. The features have been extracted through a bag of words ( Document Term Matrix ) with Tf-Idf [I don't think taking into account the length of a text, punctuation or others is relevant] - [It is mainly professional emails]

$\endgroup$
4
  • $\begingroup$ Can you provide more details - how do you extract features from the text, what classifier are you using,..? $\endgroup$ – stmax Jun 28 '16 at 13:51
  • $\begingroup$ F1 score punishes false negatives and false positives equally. So, if your classes are not uniformly distributed, you should train with equalizing weights. In sklearn, this would mean using class_weight=balanced for whatever model you are using. $\endgroup$ – Ricardo Magalhães Cruz Jun 30 '16 at 16:53
  • $\begingroup$ I am curious about what are the labels of the classification. More problem specific solution might be proposed if the problem is fully described. $\endgroup$ – William Jul 1 '16 at 19:46
  • $\begingroup$ The labels are basically two things: "Requests" and "Incident" about some softwares $\endgroup$ – bouritosse Jul 4 '16 at 8:00
3
$\begingroup$

Have you tried without the Tf-Idf weighting? What about a bi-gram analysis?

You can also review the pre processing, maybe without removing the low frequency words (wich the tf idf already does). Another idea is to remove one very high frequency word that appear on your corpora.

$\endgroup$
1
  • $\begingroup$ No didn't try without Tf-idf or bi-gram, I'm gonna try thanks. And I'm gonna remove less low frequency words that's a good idea as well $\endgroup$ – bouritosse Jul 4 '16 at 7:58
2
$\begingroup$

Feature engineering is usually the way to improve the performance.

Another possible way to improve result is through supervised feature selection. I could recommend Chi-square feature selection, step-wise feature selection. As for a more complete survey on feature selection, you could refer to Guyon, Isabelle, and André Elisseeff. "An introduction to variable and feature selection." Journal of machine learning research 3.Mar (2003): 1157-1182.

$\endgroup$
1
  • $\begingroup$ I don't know if it is really necessary I'm using the caret package and if I'm not wrong the feature selection is already done by the Train() function $\endgroup$ – bouritosse Jul 4 '16 at 7:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.