I am working on a multi-label text classification problem(Total target labels 90). The data distribution has a long tail and class imbalance and around 1900k records. Currently, I am working on a small sample of around 100k records with similar target distribution.I am using the OAA strategy (One against all). I have tried many algorithms on data.

Currently, each label has atleast 5000 data rows. The class imbalance is high with around 80k records for the most common label and the most rare with just one data row which I have not considered in the modelling. This dataset contains text from academic journals. It has Title and Abstract columns.

I am using HashingVectorizer(number of features 2**20, char analyzer) to generate features and TSVD to reduce the dimensionality(n_components=200).

LinearSVC(class_weight='balanced') # Got many warnings that it could not converge. I came to know that it may due to data not scaled properly. How can I scale text data??  
LogisticRegression(solver='lbfgs') # Converged very quickly
RandomForestClassifier(n_estimators=40,class_weight="balanced") # Train time ~2hr 

I noticed that LinearSVC has good recall(less false negatives) while Logistic and RF has good precision(less false positives) scores. Can anyone help me in identifying the reasons behind these scores and how can I improve them.

enter image description here

Currently, I am not using deep learning/transformer models due to limited computation resources.

  • 1
    $\begingroup$ 1) can you give more info on your class imbalance? E.g. a list of how many records per class. 2) What kind of text data are you dealing with? Social media? How long are records on average? $\endgroup$ – Bruno Lubascher Apr 21 '20 at 9:16
  • $\begingroup$ this kaggle compitation might help you & this blog upto an extent $\endgroup$ – Kalsi Apr 21 '20 at 9:34
  • $\begingroup$ @BrunoLubascher I have updated the post. $\endgroup$ – joel Apr 21 '20 at 13:18
  • $\begingroup$ @Kalsi Thanks. I am looking into it. $\endgroup$ – joel Apr 21 '20 at 13:19
  • $\begingroup$ @joel last question. How did you calculate your F1/P/R? I ask because you say that your most common label has 80k records out of the 100k total records. So, a simple classifier that always predict the top label should have a micro average of 0.8 F1. $\endgroup$ – Bruno Lubascher Apr 21 '20 at 14:15

As the data is imbalanced and skewed towards few classes, that's why RF and Logistic results are biased causing high FP values, therefore high precision and low recall.
SVC on the other hand might have tried to create hyperplanes to get most out of the other side of curve, thereby producing different results.

To improve the results, try to use non linear kernels and also try to balance the input data (by scaling etc) before feeding to classifiers.

  • $\begingroup$ Thanks for looking into this issue. The argument class_weight='balanced' was used wherever available. I was looking into feature scaling and came to know that TF IDF features should be passed as raw and no scaling should be done. I preferred to use HashingVectorizer to create features which will scale well but does n`t have IDF which provide feature scaling. $\endgroup$ – joel Apr 26 '20 at 14:22
  • $\begingroup$ carry out the scaling at input level itself to get balanced number of samples across classes. TF IDF vectorizer would do adjustments itself. also, did you give a try to MultinomialNaiveBayes ? $\endgroup$ – Sandeep Bhutani Apr 26 '20 at 15:56

Classifier Analysis

Try to leverage this python library for more insights: https://github.com/marcotcr/lime reference: https://marcotcr.github.io/lime/tutorials/Lime%20-%20basic%20usage%2C%20two%20class%20case.html

Deep Learning Specifics

For deep learning models, you can use Google Collab if possible.

If the model is deep learning based and is built on allennlp, the Gradient visualization can be performed. Refer this link for Demo on Sentiment Analysis Problem, https://demo.allennlp.org/sentiment-analysis/MTc0MzQwOA== Tutorial is at https://allennlp-course.apps.allenai.org/.

Multi Label Specifics

Refer here https://scikit-learn.org/stable/modules/multiclass.html.

Looks the probelm you are addressing is Multi Label Classification.

This will give you class wise reports, sklearn.metrics.classification_report. Accordingly, the narrow down poorly performing classes. And then tweak the model.

There is Micro/Macro metric for Multi Label problems. Refer here Micro Average vs Macro average Performance in a Multiclass classification setting

Suggestion My suggestion is better to explore how the metric is calculated. Find the fact that what F1 is being referred here.

Analyse F1, precision and recall, over labels. Then, it can be found out which labels are performing poorly.


You could also inform the model of the imbalance itself (either a True/False or a "class weight") depending on which modelling method you are using.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.