Deep network not able to learn imbalanced data beyond the dominant class

Question

I have data with 5 output classes. The training data has the following no of samples for these 5 classes: [706326, 32211, 2856, 3050, 901]

I am using the following keras (tf.keras) code:

class_weights = class_weight.compute_class_weight('balanced',
                                                 np.unique(y_train),
                                                 y_train)

model = tf.keras.Sequential([
    tf.keras.layers.Dense(50, input_shape=(dataX.shape[1],)),
    tf.keras.layers.Dropout(rate = 0.5),
    tf.keras.layers.Dense(50, activation=tf.nn.relu),
    tf.keras.layers.Dropout(rate = 0.5),
    tf.keras.layers.Dense(50, activation=tf.nn.relu),
    tf.keras.layers.Dropout(rate = 0.5),
    tf.keras.layers.Dense(50, activation=tf.nn.relu),
    tf.keras.layers.Dropout(rate = 0.5),
    tf.keras.layers.Dense(5, activation=tf.nn.softmax) ])
     adam = tf.keras.optimizers.Adam(lr=0.5)

model.compile(optimizer=adam, 
              loss='sparse_categorical_crossentropy',
              metrics=[metrics.sparse_categorical_accuracy])    
     model.fit(X_train,y_train, epochs=5, batch_size=32, class_weight=class_weights)

y_pred = np.argmax(model.predict(X_test), axis=1)

The first line on class_weight is taken from one of the answers in to this question: How to set class weights for imbalanced classes in Keras?

I know about this answer: Multi-class neural net always predicting 1 class after optimization . The difference is that in that case, the class weights wasn't used whereas I am using it.

I am using sparse_categorical_crossentropy which accepts categories as integers (don't need to convert them to one-hot encoding), but I also tried categorical_crossentropy and still the same problem.

I have of course tried different learning rate, batch_size, no of epochs, optimizer, and depth/length of the network. But it always is stuck at ~0.94 accuracy which is essentially I would get if I predict the first class all the time.

Not sure what is missing here. Any error? Or should I use some other specialized deep network?

Answer 1

1) A five-layer neural network is one heck of a complex model for a data set with less than 1 million points. (I’m trying to find a good link for this, but the intuition is that your choice of model should be driven by the complexity of the available data, and not by what you think the real target function is like.) If this is for a real-world project, a tool like XGBoost might work better on this data set — out of the box, you’ll spend less time dealing with problems related to imbalanced classes, poorly scaled data, or outliers. Of course if this is specifically for learning about neural networks, that advice isn’t much help!

2) For a class distribution that’s as skewed as your data, you might get more mileage from re-sampling the training data rather than re-weighting the classes during training. Down-sample the majority class first (just throw away majority samples at random); if that’s not satisfactory then try something more complicated like using SMOTE to up-sample the minority classes. Try taking this to the extreme; build a (collection of) new training sets by randomly sampling only 1,000 points from each class.

The intuition here is that, for neural networks, as far as I know, re-weighting classes basically means re-scaling the gradient steps during training based on the class weights. If the classes are skewed 10:1, that makes sense: we take a step that’s 10 times as far for a minority sample. If the classes are skewed 1000:1, as in your case, it makes less sense — we’ll take 1,000 small steps as we optimize on the majority class, and then a single gigantic step in an essentially random direction when we happen to see a minority sample, followed by 1,000 tiny steps trying to un-do this work, etc. We don’t see enough minority samples to allow information about their class to average out.

Answer 2

One important thing to check is if by any chance you have NaNs in the input data.

Had the same problem, turned out NaN's had crept in the input, working great now!

Answer 3

Is your input data standardised? What happens when you run it with only one hidden layer? What happens when you set learning rate to 1e-6 or 1e-5? What's your result when you run this through a logistic regression? What does the confusion matrix look like?

When you are using class_weights= you should also use weighted_metrics= for your convenience.

Answer 4

Why don't you try with gradient boosting or adaBoost? They should perform well in unbalanced data as, during the training, they tend to give weights to misclassified observations, improving then the performance. Lemme know

Answer 5

I think it is probably doing something, just not enough to change the overall classification. Have you inspected the estimated probability distributions of the minority class test examples for different class_weights? I imagine that the probabilities of the true classes are somewhat higher for these examples, even though the predicted most likely class is still the majority class.

If this is the case, then you can determine a value (through cross-validation or estimated on a held-out set) to subtract from the element of the estimated probability distribution corresponding to the top class before taking an argmax. i.e. test a range of values and pick the one that provides the best $F_1$ score or whatever else you want to optimise. Something like this:

# train neural net
# ... 

y_proba = model.predict(X_val)
best_score = 0.
best_value = 0.

# try every value from 0. to 1., in increments of 0.01
for i in np.linspace(0., 1., 101):
    alt_y_proba = y_proba - np.array([i, 0., 0., 0., 0.])
    alt_y_proba = np.clip(alt_y_proba, 0., 1.) # ensure no negative values!

    y_pred = np.argmax(alt_y_proba, axis=1)
    score = # some score function goes here

    if score > best_score:
        best_score = score
        best_value = i

print(best_value)    

Yes, it's a little hacky, but it may give you good results. You can re-normalise alt_y_proba to be a proper probability distribution again if you want, but it won't change the classification. Just make sure that the dataset you're optimising this subtraction value with is not used in the training of the neural net, as this may introduce overfitting.

Answer 6

I just figured out the same issue. Try to pass a dictionary like:

{
   0: 1.0, 
   1: 10, 
   2: 20, 
   3: 20, 
   4: 20
}

to class_weight in model.fit() and it will solve the problem. Understood that it says a list will also work in the docs - but seems like it does not work as well for me.