I just started training a MLP model on a data set which has the following statistics. Notice that both train and validation sets are unbalanced (88.4% negatives).
Note: The unbalance isn't caused by sampling and is because that's the natural ratio of the classes. E.g. when I look back in a 3 month period, I see I have eaten 88 apples and 12 oranges; and looking forward, I might still favor apples more than oranges by the same ratio. I may be wrong here but I think I shouldn't give class weights nor resample to balance the two classes, because that's their natural ratio.
Train data: Shape (features): (891473, 122) Shape (labels): (891473,) # of classes: [(0, 788118), (1, 103355)] % of classes: [(0, 0.88406266931247501), (1, 0.11593733068752503)] Validation data: Shape (features): (251141, 122) Shape (labels): (251141,) # of classes: [(0, 222009), (1, 29132)] % of classes: [(0, 0.8840014175303913), (1, 0.1159985824696087)]
I'm using three 8-sized dense layers each following a ReLU layer, finally a sigmoid since it's binary classification.
However, the training process converged after only 2 epochs and basically only learned the bias term and and ended up predicting everything to be negative. The confusion matrix stays unchanged like this since the 2nd epoch till the end (20 epochs):
Train confusion matrix: P = 0 P = 1 | Total L = 0 0.884 0.000 | 0.884 L = 1 0.116 0.000 | 0.116 ---------------------------- Total 1.000 0.000 | 0.884 Validation confusion matrix: P = 0 P = 1 | Total L = 0 0.884 0.000 | 0.884 L = 1 0.116 0.000 | 0.116 ---------------------------- Total 1.000 0.000 | 0.884
I have tried to increase the size of the dense layer (e.g. from 32, 128, 512, 1024). They all yielded the same result.
I'd expect that with 3 x 1024 neurons the model would be complex enough to learn something beyond a single bias term. So I'm quite confused as what I am doing wrong to make it not learning anything other than the bias.
- What could be the possible reasons for the model to learn not much?
- What should I try and/or what kind of info should I be looking for at this point to move this forward?
Thank you for your time in advance~!
The following is a snippet of my code to show what I was trying to do:
learning_rate = 1e-4 batch_size = 10000 epochs = 5 DENSE_SIZE = 1024 train = (training_features.values, training_labels.values) validation = (holdout_features.values, holdout_labels.values) test = (eval_features.values, eval_labels.values) eval_sets = [ ('Train', train), ('Validation', validation), ('Test', test), ] model = Sequential() model.add(Dense(DENSE_SIZE, activation='relu', input_shape=train.shape[1:])) model.add(Dense(DENSE_SIZE, activation='relu')) model.add(Dense(DENSE_SIZE, activation='relu')) model.add(Dense(1, activation='sigmoid')) # initiate optimizer opt = RMSprop(lr=learning_rate) # Let's train the model model.compile(loss='binary_crossentropy', optimizer=opt, metrics=['accuracy']) model.fit( train, train, batch_size=batch_size, epochs=epochs, validation_data=validation, shuffle=True, verbose=2, callbacks=callbacks, )
The three answers as of writing all suggest that the inefficiency in learning is associated w/ the imbalanced classes and I should try to balance the classes by either using class weights or resampling.
While I'm happy to see an apparent agreement of what the problem is what , could anyone provide more insights about why imbalanced classes cause difficulty even for a complex model? In theory a complex enough model can at least memorize all the rarely positive samples and achieve a higher-than-bias accuracy on the training set.