I have created a synthetic dataset, with 20 samples in one class and 100 in the other, thus creating an imbalanced dataset. Now the accuracy of classification of the data before balancing is 80% while after balancing (i.e., 100 samples in both the classes) it is 60%. What are the possible reasons for this?
$\begingroup$
$\endgroup$
1
-
$\begingroup$ What and how much was contributed to the model (trained) dataset of 20 samples and 100 samples? $\endgroup$– gurvinder372Commented Feb 23, 2018 at 9:04
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
1
Imagine that your data is not easily separable. Your classifier isn't able to do a very good job at distinguishing between positive and negative examples, so it usually predicts the majority class for any example. In the unbalanced case, it will get 100 examples correct and 20 wrong, resulting in a 100/120 = 83% accuracy. But after balancing the classes, the best possible result is about 50%.
The problem here is that accuracy is not a good measure of performance on unbalanced classes. It may be that your data is too difficult, or the capacity of your classifier is not strong enough. It's usually better to look at the confusion matrix to better understand how the classifier is working, or look at metrics other than accuracy such as the precision and recall, $F_1$ score (which is just the harmonic mean of precision and recall), or AUC. These are typically all easy to use in common machine learning libraries like scikit-learn.
-
$\begingroup$ the accuracy is reported on real test set, not synthetic test set. $\endgroup$– keramatCommented Nov 3, 2020 at 20:32
$\begingroup$
$\endgroup$
Accuracy is probably not a good metric for your problem.
For the original dataset, if the model just makes a dummy prediction that all samples belong to the bigger class, the accuracy will be 83% (100/120). But that's usually not what we want to predict in an imbalanced dataset.
Let's take a fraud detection problem. The probability that a transaction is a fraud is very small (let's say 0.01%) but the loss of an undetected fraud transaction is enormous (e.x. 1 millions dollars). On the other hand, the cost of manually verifying if a transaction is relatively small. In that case, we would like to detect all possible frauds, even if we have to make a lot of false positive predictions.
To tackle an imbalanced dataset, first you have to choose which question you want to answer. Then, what's the good metric for this question. Answer these 2 question first before deciding which technique you should use.
Come back to the original question. Why does accuracy reduce when we oversample the smaller class?
That's because this technique puts more weight to the small class, makes the model bias to it. The model will now predict the small class with higher accuracy but the overall accuracy will decrease.
