0
$\begingroup$

Dear DataScience Community, I am working on class imbalance tabular data with high-dimension inputs. The tabular data is derived from the satellite data pixels, and I have inflated the train data dimension with derived indices calculated from available data. The accuracy measured with the F1 score is very low for the classes with low data counts. I've attached the sample data size.

enter image description here

I used MLP, CatBoostClassifier, and SVM for the task, with CatBoostClassifier giving me the best result. To address the data classification accuracy, I implemented oversampling, undersampling(losing a large chunk of features of classes with higher count), SMOTE, GSMOTE, and an algorithmic approach where I assigned weight during the model training part in CatBoostClassifier with class_weights parameter. The best result is when I used the algorithmic approach assigning the class_weights.

F1 Metrics

Still, the class with higher samples has higher F1 scores, and the performance of my model is not good in the undersampled classes. Gone through a lot of literature but didn't come to a valid conclusion on a better approach to deal with this level of imbalance and on classification model and evaluation metric selection too.

Next, I am planning to divide the data into chunks and train on a different model, and use a voting approach to classify and update the status after completion.

So, I would love to hear how the datascience community has tackled the situation to deal with this kind of data imbalance. Also, idea of any deep learning method suitable for the challenge?

$\endgroup$
4
  • $\begingroup$ Instead of choosing the class with the highest confidence score, you could use some threshold for your poor performing classes. You will have to write some custom logic to deal with cases like multiple classes being above the threshold. $\endgroup$ Sep 13 at 6:52
  • $\begingroup$ @MichaelHiggins, could you please explain more or provide some reference on how a threshold can be manually set for underrepresented classes? $\endgroup$ Sep 13 at 7:05
  • $\begingroup$ I have only seen class specific thresholding for multi-label problems. I think it's worth a shot though. When multiple classes are above their threshold, you need some method to choose between them. I would try picking the rarest one. Or some function of rarity+how far above the threshold you are. $\endgroup$ Sep 13 at 18:58

1 Answer 1

2
$\begingroup$

I have implemented something at work on the dataset which has so many classes as above and it worked well for me, though generalization might be an issue but its worth trying, what I would do in your case is initially have a model which would predict class A, class B and Class C-M combined(you can call it class Special) so it will be a 3 class problem, once you have good accuracy here you can create another model which would predict the independent class from the class special. Again I am telling you this worked for me at my work and the model is deployed in production too.

$\endgroup$
2
  • 1
    $\begingroup$ Yeah, I am planning to do the same as the next step. Do you have a code sample or resource you can point to for implementing such a collection of models like in decision tree fashion? $\endgroup$ Sep 15 at 3:20
  • $\begingroup$ This would work well with the alteration of using binary classifiers, and altering the threshold til you get desired performance on the "other" class. $\endgroup$ Sep 16 at 6:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.