0
$\begingroup$

I have an imbalanced dataset. My classes are 0 and 1. The number of 0 class instances is about 20 times more than the number 1 class instances. I know that I should apply oversampling after train test split (to not having over-optimistic results.) and it is better to have original data in the test set, not synthetic.

But there is something that I don't understand how to deal with it. Let's say:

  • I have a data file of 525 lines. 25 lines belong to class 1, 500 lines belong to class 0.
  • I apply 5-fold cross-validation.
  • At each time, I split 420 lines for train and 105 lines for the test. Then I oversample train data, so my train data becomes balanced (with an equal number of 0 and 1 classes).
  • But in my test data (105 lines), I have 5 instances of class 1, 100 instances of class 0. After doing classification using training model and test data, I see the imbalance here and I cannot interpret the results (confusion matrix, accuracy, f1-score, tp, fp etc.).

I need to test more instance of class 1. I am not able to interpret results with a class rate of like 5/100.

Does anyone have any idea how to do it? Thank you.

$\endgroup$
0
$\begingroup$

Thing is you don’t have 5/100, you have 25/500. You don’t evaluate each fold separately, you evaluate them together. And if 25 positive cases are not enough for you to feel like you can properly evaluate your model, well then you have to go get more data because no amount of under/over sampling will fix that.

| improve this answer | |
$\endgroup$
0
$\begingroup$

Depending on the type of algorithm you're using, you might use logloss as your optimization metric - because it will eliminate part of the problem you're trying to avoid by having such an imbalance.

I often find that the effort of under/over sampling the data does not yield better predictive performance than just using logloss from the beginning... until your % gets super low, or you're running into memory issues on the computer.

| improve this answer | |
$\endgroup$
0
$\begingroup$

You have two challenges - Imbalanced class and small sample size.

Best that can be done -
- Have ~8 sample as test [This means, set K = 3]
Still your minor class accuracy will drop by ~14% with each misclassification.

- Add downsampling in addition to upsampling

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.