4
$\begingroup$

I want to use class weights for training a CNN with a imbalanced data set. The question arise if the sum of the weights of all examples have to stays the same?

My previous plan was to use the function compute_class_weight('balanced,np.unique(y_train),y_train) function from scikit-learn.

But I'm totally unsure if this is even suitable for the class weights of a CNN?

Thank you in advance for each tip

Improve this question
$\endgroup$
3
  • $\begingroup$ any reason why would you claim that? $\endgroup$
    – Yohanes Alfredo
    Jan 10, 2020 at 15:44
  • 1
    $\begingroup$ @Yohanes Alfredo I'm a beginner in machine learning and I therefore wonder if this is really the right approach. I have lack experience in dealing with class weights. I want to set class weights because I also want to use EarlyStopping and therefore I try to balance the data set with class weights. But I don't know if this is the right approach. $\endgroup$
    – Code Now
    Jan 10, 2020 at 15:49
  • $\begingroup$ Related question: datascience.stackexchange.com/questions/13490/… $\endgroup$
    – Akavall
    Jan 10, 2020 at 16:26

1 Answer 1

Reset to default
4
$\begingroup$

If the "cost" for experimenting is not really that big I suggest you take the time to experiment and take this as a learning opportunity and just try if it could actually work.

There are many approaches to address class imbalance and setting class weight is one of them and the easiest to implement.

  • Change loss function (for example to focal loss for binary classification with extreme imbalance)
  • Oversampling and Undersampling
  • Setting class weights
  • Use specific algorithm that are build to address this problem e.g. siamese network which is very useful when you only say have very few training sample of object of interest.
  • etc.

Specifically for your case, I can tell you the specific case that it could fail base on my experience. So basically this very likely fail when you have extreme class imbalance say like 1% positive and 99% negative. How this could fail is simply because using class weighting in this case will put very high value on the positive sample and if your model fails to detect this, the penalty is very high and hence lead to unstable training. To top it off consider a hypothetical situation your model predict the positive class correctly on epoch 10 and then it fails on epoch 11. For this case you might get a loss for example 1.3 for epoch 10 but then on epoch 11 your loss could go to say like 37.7 simply because it fail to detect said sample. This could also affect any callbacks that utilize this loss.

In summary if the situation could be as I described then don't use this otherwise just play around and find out what's best for you.

Improve this answer
$\endgroup$
5
  • $\begingroup$ @ Yohanes Alfred It is the gtsrb data set with 43 classes. The smallest class contains 210 images, the largest class about 2000 images. The remaining classes are in between. No class should be preferred. All classes should simply be considered to be equal important. $\endgroup$
    – Code Now
    Jan 10, 2020 at 17:27
  • $\begingroup$ If that is the case I suggest doing oversampling but no need to use library, you can try my approach. First, set predefined steps of training for each epoch. For each batch you sample with equal probability which classes will appear for that batch. And then sample the set amount as required. For example, if according to your first sampling your batch will have 10 occurence of class 1, then you go on the list of image which has class 1 and sample 10 image. This kind of guarantees that they are sampled with equally while maintaining randomness/shuffling. $\endgroup$
    – Yohanes Alfredo
    Jan 10, 2020 at 18:17
  • $\begingroup$ If you need more hint, I can link you a notebook of mine which I just play around with the method I mentioned. $\endgroup$
    – Yohanes Alfredo
    Jan 10, 2020 at 18:18
  • $\begingroup$ My case is a little more complicated. I would also like to train SVMs with GTSRB, both with the entire data set and only with subsets of it. The same is said to be done with CNNs. In the end I want to compare the results with each other, i.e. at which subset SVM or CNN is better. For the SVMs 'class_weight='balanced'' works well. The problem is, that I have to optimize the hyperparameters from the SVMs through GridSearchCV before training with the data set. When I do oversampling and after that I optimize the hyperparameters via gridSearch, this would take a very long time. $\endgroup$
    – Code Now
    Jan 10, 2020 at 18:47
  • $\begingroup$ Oversampling will not take more time, despite imagining that oversampling will increase your number of data, but at the end of the day what matters when your model converge. Say if you train the model without oversampling and it converges in 2000 steps, then with oversampling it will also converge in roughly similar number of steps. You can try on your own if you don't believe me. $\endgroup$
    – Yohanes Alfredo
    Jan 11, 2020 at 2:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.