Text Classification on a very small data set with a lot of classes

Question

I have a data set consisting of 455 rows spread over 21 different classes. The data set is imbalanced as well as you can see below.

job_alerts                    45
howto_apply                   40
application_status            31
salary                        30
job_close_date                30
multiple_role                 27
feedback                      26
assessment_campatilibility    24
interview_reschedule          23
disability                    22
reinstate_application         19
job_account_issue             19
assessment_link_problem       16
age_limit                     16
assessment_timebox            16
cv_past_experience            15
late_for_interview            13
interview_response_time       13
work_experience               11
assessment_validity           10
special_needs_at_work          9

I'm trying to perform classification on this. Firstly, when preprocessing the data I convert the text to lowercase, remove stopwords and punctuation and convert words to their lemmas using spaCy.

So far, I have tried the TextCategorizer in spaCy which uses a simple CNN for classification and I'm getting around 70% accuracy on an 80-20 split of the data. I have also tried various Sklearn algorithms such as SVC, LogisticRegression, RandomForestClassifier and MultinomialDB with the TfIDVectorizer. I have used GridSearchCV to tune the parameters. The best results I get are with SVC at around 74-75% accuracy.

I want to know what else I can try to improve my accuracy. I am a beginner to NLP and I haven't worked on something like this before. Right now the ideas I have are to improve the TfIdVectorizer through some parameter tuning and to try a One vs all classifier for SVC. What else can you suggest to me?

Answer 1

Accuracy is not the best measure for imbalanced data. Prefer precision and recall. Do undersampling/oversampling to get equal samples for each class and try XGBoost. Or else you can use SVC with class weights, give lower class weight to classes with more samples and vice versa.