Suppose we have a game and its action space contains two possible actions: A and B.

We have a labelled dataset of state-action pairs but 95% of actions are A and only 5% are B

If we train a neural network, it will always output A as it will choose the most probable class to decrease its loss.

Are there ways to solve this problem?

  • 3
    $\begingroup$ 1) Are you using reinforcement learning or are you training a classifier? 2) You claim the network will always output A. Do you have any evidence that it is happening or are you speculating? Indeed, when you deal with class imbalance, it is likely that the majority class will be chosen more often, but the minority class definitely has a chance to be predicted as well. $\endgroup$ Commented May 7, 2020 at 11:43
  • 1
    $\begingroup$ One approach would be to "oversample" the minority class, e.g. for each sample from A that is shown to the network, 10 samples from Bare shown. $\endgroup$
    – Peter
    Commented May 7, 2020 at 12:07
  • $\begingroup$ @ValentinCalomme For a classifier we can split our data and make a balance between two classes but if we have RL problem it is harder to split the data. suppose we have a continuous q-table and we can't manipulate it. can we use a custom loss function that it is more sensitive to B or using different network architecture. I trained a network on such a problem like this and it's output always was the first class. i just speculate it that network consider the second class as noise. $\endgroup$
    – amin msh
    Commented May 7, 2020 at 20:03
  • $\begingroup$ What type of neural network are you using? $\endgroup$
    – Tank
    Commented May 14, 2020 at 11:07

4 Answers 4


This is for classification, and I am not sure if it is possible to extend them to reinforcement learning.

As you figured out, accuracy should not be used as a metric for a dataset as imbalanced as the one you have. Instead, you should look at a metric such as Area Under Curve(AUC). If you would have infinite data, then you could just rebalance and remove some of the data from the class that has the most samples. However, in many cases data is sparse and you want to use as much of it as possible. Removing data can have a disastrous effect on many applications.

So what are good and convenient ways of handling this?

  • Add weights to the loss function. One weight for class A and one for B. By increasing the magnitude of the loss for the B class the model should not get stuck in a suboptimal solution that just predicts one class.

  • Use another objective(loss) function. F1-score can, for example, be implemented and used as an objective(loss) function. For a differentiable version see https://stackoverflow.com/a/65320239/12229416

What is great with these approaches is that it will allow you to use all the data.

  • $\begingroup$ Thank you for the answer. I have one question, can i use conditional loss function instead of weighted loss function? for example use two loss function and check the the true output and choose the loss for this case with if statement. $\endgroup$
    – amin msh
    Commented May 12, 2020 at 11:21
  • $\begingroup$ Suppose there is three possible actions A, B and C , if we always output A the loss doesn't change a lot but by choosing B and C make the loss more sensitive to true or wrong prediction. $\endgroup$
    – amin msh
    Commented May 12, 2020 at 11:26
  • $\begingroup$ Yes! So then you would multiply the loss with a different weight depending on the ground truth label. $\endgroup$ Commented May 14, 2020 at 12:04
  • $\begingroup$ @CarlRynegardh "F1-score can, for example, be implemented and used as an objective(loss) function". This isn't true. F1-score is a non-differentiable function so I don't see how you could use it as an objective/loss function in a neural network. -1 $\endgroup$
    – ATP
    Commented Jun 13, 2021 at 17:52
  • $\begingroup$ @ATP it can be approximated (but still a bad idea to use this). $\endgroup$ Commented Dec 19, 2023 at 9:27

You have tagged the question with reinforcement-learning, but you describe a labeled dataset, suggesting supervised learning. I will try to cover both cases.

There are some techniques that are applicable in both supervised learning and reinforcement learning:

  • Sample from the buffer conditioning on the action, to have a balanced dataset regarding action taken.
  • Apply data augmentation techniques on the minority action class. The Synthetic Minority Oversampling TEchnique (SMOTE) algorithm may be an option for that. The problem with data augmentation is that you would need to do it in the RL loop, which can enlarge the needed computation time.

Note that to apply them for reinforcement learning, you should use a replay buffer, like they do in the DeepMind Atari paper.

If you are in a supervised learning scenario, you can apply class weights, e.g. this example in Keras.


As people have mentioned above you want to try and up-sample / bootstrap. In other words you want to try and get the classes to have similar proportions. One way to do this is to simply randomly select the less likely sample.

More complicated solutions: 1. involve adding realistic noise to the less likely class to increase the number of data points. 2. Using a different score/error function - look at balanced accuracy 3. Initiate the training with 50% A and 50% B - once it converges start training it gradually on a larger part of the data set which will gradually become 95% A and 5% B.


For Imbalanced classes, the method which I prefer the most is bootstrapping.

  1. Lets say you have n classes with number of examples as m , 2m, 3m (this is just to tell which is the minimum).

  2. create multiple dataset with m samples from each classes. (randomly)

  3. keep training on each one of them .


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.