2
$\begingroup$

I just started training a MLP model on a data set which has the following statistics. Notice that both train and validation sets are unbalanced (88.4% negatives).

Note: The unbalance isn't caused by sampling and is because that's the natural ratio of the classes. E.g. when I look back in a 3 month period, I see I have eaten 88 apples and 12 oranges; and looking forward, I might still favor apples more than oranges by the same ratio. I may be wrong here but I think I shouldn't give class weights nor resample to balance the two classes, because that's their natural ratio.

Train data:
    Shape (features): (891473, 122)
    Shape (labels): (891473,)
    # of classes: [(0, 788118), (1, 103355)]
    % of classes: [(0, 0.88406266931247501), (1, 0.11593733068752503)]
Validation data:
    Shape (features): (251141, 122)
    Shape (labels): (251141,)
    # of classes: [(0, 222009), (1, 29132)]
    % of classes: [(0, 0.8840014175303913), (1, 0.1159985824696087)]

I'm using three 8-sized dense layers each following a ReLU layer, finally a sigmoid since it's binary classification.

However, the training process converged after only 2 epochs and basically only learned the bias term and and ended up predicting everything to be negative. The confusion matrix stays unchanged like this since the 2nd epoch till the end (20 epochs):

Train confusion matrix:
           P = 0   P = 1 | Total
    L = 0  0.884   0.000 | 0.884
    L = 1  0.116   0.000 | 0.116
    ----------------------------
    Total  1.000   0.000 | 0.884
Validation confusion matrix:
           P = 0   P = 1 | Total
    L = 0  0.884   0.000 | 0.884
    L = 1  0.116   0.000 | 0.116
    ----------------------------
    Total  1.000   0.000 | 0.884

I have tried to increase the size of the dense layer (e.g. from 32, 128, 512, 1024). They all yielded the same result.

I'd expect that with 3 x 1024 neurons the model would be complex enough to learn something beyond a single bias term. So I'm quite confused as what I am doing wrong to make it not learning anything other than the bias.

Question:

  1. What could be the possible reasons for the model to learn not much?
  2. What should I try and/or what kind of info should I be looking for at this point to move this forward?

Thank you for your time in advance~!

The following is a snippet of my code to show what I was trying to do:

learning_rate = 1e-4
batch_size = 10000
epochs = 5

DENSE_SIZE = 1024

train = (training_features.values, training_labels.values)
validation = (holdout_features.values, holdout_labels.values)
test = (eval_features.values, eval_labels.values)
eval_sets = [
    ('Train', train),
    ('Validation', validation),
    ('Test', test),
]

model = Sequential()
model.add(Dense(DENSE_SIZE, activation='relu', input_shape=train[0].shape[1:]))
model.add(Dense(DENSE_SIZE, activation='relu'))
model.add(Dense(DENSE_SIZE, activation='relu'))
model.add(Dense(1, activation='sigmoid'))

# initiate optimizer
opt = RMSprop(lr=learning_rate)

# Let's train the model
model.compile(loss='binary_crossentropy', optimizer=opt, metrics=['accuracy'])

model.fit(
    train[0],
    train[1],
    batch_size=batch_size,
    epochs=epochs,
    validation_data=validation,
    shuffle=True,
    verbose=2,
    callbacks=callbacks,
)

EDIT:

The three answers as of writing all suggest that the inefficiency in learning is associated w/ the imbalanced classes and I should try to balance the classes by either using class weights or resampling.

While I'm happy to see an apparent agreement of what the problem is what , could anyone provide more insights about why imbalanced classes cause difficulty even for a complex model? In theory a complex enough model can at least memorize all the rarely positive samples and achieve a higher-than-bias accuracy on the training set.

$\endgroup$
  • $\begingroup$ I found the following article helpful on this topic. It talks about not only the different ways of attacking the imbalanced classes, but also the reasons behind. svds.com/learning-imbalanced-classes $\endgroup$ – Roy May 31 '17 at 5:38
2
$\begingroup$

Generally there are a number of ways to deal with class imbalance (more details here fore example).

Specifically in your case, I would either try training with the training samples evenly distributed in classes or I would play around changing the loss function, to take into account the different classes' support. For the latter, you can check this discussion also .

| improve this answer | |
$\endgroup$
  • $\begingroup$ Marking this as the accepted answer because this is the only one which mentions both resampling and using class weights as solutions. $\endgroup$ – Roy May 31 '17 at 5:25
2
$\begingroup$

Though you know that dataset is naturally balanced but your model will still consider it as unbalanced dataset. If you take a example financial fraud detection, number of fraudulent transactions are always less than the ligit translations, to handle such such cases data scientists will have to balance the classes for better results. You will have to balance your dataset to get the better results from model.

| improve this answer | |
$\endgroup$
1
$\begingroup$

It is not a matter of model complexity. In order to address class imbalance, you should specify class weights as argument to the fit function in order for the logits of the cross entropy to be scaled accordingly.

You can find further info in stackoverflow and examples on how to provide the weights to fit here.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.