1
$\begingroup$

I am a researcher in Machine Learning. In my project, I have been applying ML to a small imbalanced data consisting of 8 features and 297 instances with 44 positive instances and 253 negative ones. Firstly, I split the whole dataset into a training set (80%) and a testing set (20%) using stratified sampling. Secondly, I oversampled the training set to a balanced training set using random sampling with replacement or SMOTE; and applied information gain feature selection to reduce the features of the balanced training set. Thirdly, I trained logistic regression (LR), RBF SVM, polynomial SVM, MLP (1-hidden layer networks and 2-hidden layers networks), random forest, XGBoost, the evolutionary fuzzy classifier (Weka 3.9.4), Sugeno fuzzy system (Matlab) respectively on the reduced training set. For the oversampling step, random sampling with replacement leads to better classification performance than SMOTE. The size of the balanced training set was set to 400, 600, 800, and 1000 instances respectively. After training, each classifier was tested on the test set. This training-testing process was repeated 100 times with each time using a different training set (80%) and a test set (20%).

The results of testing are not satisfactory: LR has the best average AUC of 0.58 on the test set over 100 times.

What other algorithms do I need to try to improve the performance? Much appreciated.

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ Statisticians do not see class imbalance as a problem, and there is no need to use undersampling, oversampling, or artificial balancing to solve a non-problem. It might be helpful if you say why you find the imbalance problematic. stats.stackexchange.com/questions/357466 fharrell.com/post/class-damage fharrell.com/post/classification stats.stackexchange.com/a/359936/247274 stats.stackexchange.com/questions/464636 twitter.com/f2harrell/status/1062424969366462473?lang=en $\endgroup$
    – Dave
    Commented Jan 3, 2022 at 0:17
  • $\begingroup$ The small size of data set (297 instances) and class imbalance caused the poor AUC on test set of 0.58. I have also applied the same machine learning approaches with the same parameter settings on another bigger imbalanced dataset with 402 instances (108 positive instances and 294 negative instances) and 32 features. The RBF SVM has the best average AUC of 0.63 on test set over 100 times of the training and testing process. $\endgroup$
    – David Tian
    Commented Jan 3, 2022 at 18:08
  • $\begingroup$ @DavidTian the problem you have is not with imbalance (1/6 is not real imbalance), but with the size of your data set / its separability. There is probably not much to learn... Have you tried basic EDA (2D PCA to see if your data is "separable") ? $\endgroup$
    – Lucas Morin
    Commented Jun 5, 2022 at 14:16
  • $\begingroup$ @DavidTian You need to consider a big dataset. From my personal experience, I got the worst result for 1000 instances but when I train the model with 15000 instances the accuracy, recall, and precision changed drastically. Sampling, feature selection would not help much unless you run your model with a large dataset. $\endgroup$
    – Encipher
    Commented Oct 15, 2022 at 4:41

1 Answer 1

Reset to default
-1
$\begingroup$

To handle the class imbalance, I am aware of two broad categories of techniques. The first type of technique directly solves the problem by changing the data distribution itself. On the other hand, the second type of technique plays with the loss function to solve the problem.

  1. Over Sampling and Under Sampling techniques
  2. Modifying Loss functions

I believe you have used the first type of technique and did not see much improvement. I recommend trying using the second type of technique to modify the loss function to encounter the imbalance in the dataset.

Weighted cross-entropy loss

We can assign weights to the cross-entropy loss such that it will penalize more to the smaller classes and the less to larger classes. Many frameworks have a very easy way to do this. In Scikit-learn you can look out for the class_weight parameter. For eg - random forest (https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html)

Focal loss

Originally proposed for object detection, but we can also use this for any other use case. This article (https://amaarora.github.io/2020/06/29/FocalLoss.html) succinctly explains the whole idea. However, currently, I do not know how to use this for sklearn models.

Combining both techniques

Lastly, these types of techniques are effective when the dataset is large. So,I further recommend using these techniques after SMOTE or Oversampling.

Improve this answer
$\endgroup$
7

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.