0
$\begingroup$

Dear DataScience Community, I am working on class imbalance tabular data with high-dimension inputs. The tabular data is derived from the satellite data pixels, and I have inflated the train data dimension with derived indices calculated from available data. The accuracy measured with the F1 score is very low for the classes with low data counts. I've attached the sample data size.

enter image description here

I used MLP, CatBoostClassifier, and SVM for the task, with CatBoostClassifier giving me the best result. To address the data classification accuracy, I implemented oversampling, undersampling(losing a large chunk of features of classes with higher count), SMOTE, GSMOTE, and an algorithmic approach where I assigned weight during the model training part in CatBoostClassifier with class_weights parameter. The best result is when I used the algorithmic approach assigning the class_weights.

F1 Metrics

Still, the class with higher samples has higher F1 scores, and the performance of my model is not good in the undersampled classes. Gone through a lot of literature but didn't come to a valid conclusion on a better approach to deal with this level of imbalance and on classification model and evaluation metric selection too.

Next, I am planning to divide the data into chunks and train on a different model, and use a voting approach to classify and update the status after completion.

So, I would love to hear how the datascience community has tackled the situation to deal with this kind of data imbalance. Also, idea of any deep learning method suitable for the challenge?

Improve this question
$\endgroup$
5
  • $\begingroup$ Instead of choosing the class with the highest confidence score, you could use some threshold for your poor performing classes. You will have to write some custom logic to deal with cases like multiple classes being above the threshold. $\endgroup$
    – Michael Higgins
    Commented Sep 13, 2022 at 6:52
  • $\begingroup$ @MichaelHiggins, could you please explain more or provide some reference on how a threshold can be manually set for underrepresented classes? $\endgroup$
    – hillsonghimire
    Commented Sep 13, 2022 at 7:05
  • $\begingroup$ Class imbalance and evaluation of “classification” models have a number of common misconceptions that have minimal grounding in sound statistics. $\endgroup$
    – Dave
    Commented Sep 13, 2022 at 11:13
  • $\begingroup$ I have only seen class specific thresholding for multi-label problems. I think it's worth a shot though. When multiple classes are above their threshold, you need some method to choose between them. I would try picking the rarest one. Or some function of rarity+how far above the threshold you are. $\endgroup$
    – Michael Higgins
    Commented Sep 13, 2022 at 18:58
  • $\begingroup$ @MichaelHiggins Can you explain how I can implement your idea? $\endgroup$
    – hillsonghimire
    Commented Sep 30, 2022 at 7:00

1 Answer 1

Reset to default
2
$\begingroup$

I have implemented something at work on the dataset which has so many classes as above and it worked well for me, though generalization might be an issue but its worth trying, what I would do in your case is initially have a model which would predict class A, class B and Class C-M combined(you can call it class Special) so it will be a 3 class problem, once you have good accuracy here you can create another model which would predict the independent class from the class special. Again I am telling you this worked for me at my work and the model is deployed in production too.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ Yeah, I am planning to do the same as the next step. Do you have a code sample or resource you can point to for implementing such a collection of models like in decision tree fashion? $\endgroup$
    – hillsonghimire
    Commented Sep 15, 2022 at 3:20
  • $\begingroup$ This would work well with the alteration of using binary classifiers, and altering the threshold til you get desired performance on the "other" class. $\endgroup$
    – Michael Higgins
    Commented Sep 16, 2022 at 6:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.