2
$\begingroup$

Suppose we have a game and its action space contains two possible actions: A and B.

We have a labelled dataset of state-action pairs but 95% of actions are A and only 5% are B

If we train a neural network, it will always output A as it will choose the most probable class to decrease its loss.

Are there ways to solve this problem?

$\endgroup$
4
  • 2
    $\begingroup$ 1) Are you using reinforcement learning or are you training a classifier? 2) You claim the network will always output A. Do you have any evidence that it is happening or are you speculating? Indeed, when you deal with class imbalance, it is likely that the majority class will be chosen more often, but the minority class definitely has a chance to be predicted as well. $\endgroup$ May 7 '20 at 11:43
  • 1
    $\begingroup$ One approach would be to "oversample" the minority class, e.g. for each sample from A that is shown to the network, 10 samples from Bare shown. $\endgroup$
    – Peter
    May 7 '20 at 12:07
  • $\begingroup$ @ValentinCalomme For a classifier we can split our data and make a balance between two classes but if we have RL problem it is harder to split the data. suppose we have a continuous q-table and we can't manipulate it. can we use a custom loss function that it is more sensitive to B or using different network architecture. I trained a network on such a problem like this and it's output always was the first class. i just speculate it that network consider the second class as noise. $\endgroup$
    – amin msh
    May 7 '20 at 20:03
  • $\begingroup$ What type of neural network are you using? $\endgroup$
    – Tank
    May 14 '20 at 11:07
4
+50
$\begingroup$

This is for classification, and I am not sure if it is possible to extend them to reinforcement learning.

As you figured out, accuracy should not be used as a metric for a dataset as imbalanced as the one you have. Instead, you should look at a metric such as Area Under Curve(AUC). If you would have infinite data, then you could just rebalance and remove some of the data from the class that has the most samples. However, in many cases data is sparse and you want to use as much of it as possible. Removing data can have a disastrous effect on many applications.

So what are good and convenient ways of handling this?

  • Add weights to the loss function. One weight for class A and one for B. By increasing the magnitude of the loss for the B class the model should not get stuck in a suboptimal solution that just predicts one class.

  • Use another objective(loss) function. F1-score can, for example, be implemented and used as an objective(loss) function.

What is great with these approaches is that it will allow you to use all the data.

$\endgroup$
4
  • $\begingroup$ Thank you for the answer. I have one question, can i use conditional loss function instead of weighted loss function? for example use two loss function and check the the true output and choose the loss for this case with if statement. $\endgroup$
    – amin msh
    May 12 '20 at 11:21
  • $\begingroup$ Suppose there is three possible actions A, B and C , if we always output A the loss doesn't change a lot but by choosing B and C make the loss more sensitive to true or wrong prediction. $\endgroup$
    – amin msh
    May 12 '20 at 11:26
  • $\begingroup$ Yes! So then you would multiply the loss with a different weight depending on the ground truth label. $\endgroup$ May 14 '20 at 12:04
  • $\begingroup$ @CarlRynegardh "F1-score can, for example, be implemented and used as an objective(loss) function". This isn't true. F1-score is a non-differentiable function so I don't see how you could use it as an objective/loss function in a neural network. -1 $\endgroup$
    – ATP
    Jun 13 at 17:52
2
$\begingroup$

You have tagged the question with reinforcement-learning, but you describe a labeled dataset, suggesting supervised learning. I will try to cover both cases.

There are some techniques that are applicable in both supervised learning and reinforcement learning:

  • Sample from the buffer conditioning on the action, to have a balanced dataset regarding action taken.
  • Apply data augmentation techniques on the minority action class. The Synthetic Minority Oversampling TEchnique (SMOTE) algorithm may be an option for that. The problem with data augmentation is that you would need to do it in the RL loop, which can enlarge the needed computation time.

Note that to apply them for reinforcement learning, you should use a replay buffer, like they do in the DeepMind Atari paper.

If you are in a supervised learning scenario, you can apply class weights, e.g. this example in Keras.

$\endgroup$
0
$\begingroup$

For Imbalanced classes, the method which I prefer the most is bootstrapping.

  1. Lets say you have n classes with number of examples as m , 2m, 3m (this is just to tell which is the minimum).

  2. create multiple dataset with m samples from each classes. (randomly)

  3. keep training on each one of them .

$\endgroup$
0
$\begingroup$

As people have mentioned above you want to try and up-sample / bootstrap. In other words you want to try and get the classes to have similar proportions. One way to do this is to simply randomly select the less likely sample.

More complicated solutions: 1. involve adding realistic noise to the less likely class to increase the number of data points. 2. Using a different score/error function - look at balanced accuracy 3. Initiate the training with 50% A and 50% B - once it converges start training it gradually on a larger part of the data set which will gradually become 95% A and 5% B.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.