I have a dataset that is highly imbalanced. One class has 412 (class 0) samples while the other has 67215 (class 1) samples. For its classification, I am using MLP. When I use class weight of 165 for class 0 and 1 for class 1 (in keras), I am getting extremely bad results. However, if I oversample the dataset, I am getting really good results. What is the reason behind this?
$\begingroup$
$\endgroup$
3
-
$\begingroup$ I suppose that when you say "class weight" you refer to the loss function. Keep in mind that the training is done in steps of gradient-decent. When you over-sample, you in other words make the network train in more steps and make less "dangerous" steps, while when using class weight, you kind of boost the learning rate of the class that has less samples and thus you can get the optimization algorithm out of track, as it may make too big, "dangerous" steps, based just on a few samples (or even a single one) that it has in a batch. $\endgroup$– SomethingSomethingApr 20, 2020 at 11:07
-
$\begingroup$ After all, you want to help the optimizer (usually SGD) minimize the function using gradient decent steps, and that's tricky, since there are a lot of hyper-parameters, such as learning rate and batch size. People choose these hyper-parameters based on their experience and common-sense and this is one of the main reasons why deep-learning is not already automated and still requires experienced engineers in the loop. Your question falls IMO to the same bucket of hyper-parameter problems that have no proven answer and common sense can be used retrospectively to explain why something happened. $\endgroup$– SomethingSomethingApr 20, 2020 at 11:18
-
$\begingroup$ What exactly do you mean by results? On a separate test set? Was it resampled (and how)? $\endgroup$– Ben Reiniger ♦Apr 21, 2020 at 19:19
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
2
It could be your sampling strategy
If you are oversampling by just duplicating data from class 0, then it is likely that you are overfitting. The same datapoint will be seen over and over.
You could try another oversampling strategy, for example, SMOTE or ADASYN. These techniques create data points that are closets to decision boundaries so you are less inclined to overfit on "easy" data points.
Another things you can try is oversampling the minority class and undersampling the majority class that the same time. When choosing a method to do this, pick one that can oversample near decision boundaries and undersample away from decision boundaries. For example, here is SMOTETomek. Notice how classes purple and green get mainly oversampled and class yellow mainly undersampled.
These images come from imbalanced-learn which is a Python package you can use for all these sampling strategies.
It could be your pipeline
If you use your oversampled data for testing your model performance, you could be (unwillingly) manipulating your results. You need to ensure that you use your augmented data only for training, and not for validation and testing.
+-> training set ---> data augmentation --+
| |
| +-> model training --+
| | |
all data -+-> validation set -----------------------+ |
| +-> model testing
| |
| |
+-> test set --------------------------------------------------+
answered Apr 19, 2020 at 5:24
-
1$\begingroup$ Even though the user asked why oversampling works better than using biased loss function, this answer is really interesting, it's good to know about these things, I haven't known them before $\endgroup$ Apr 20, 2020 at 15:22
-
$\begingroup$ I agree with @SomethingSomething. It is indeed a good explanation $\endgroup$ Apr 25, 2020 at 5:57