So I am training a neural network on a binary classification problem and my Case (1) and Controls (0) were imbalanced so I oversampled my cases so that that the training set was 0.5053 made up of controls. I did not balance my test set which was 0.562 controls. In the beginning my train and test accuracy rises (it is not very accurate still but I expect this to be the case) but then the training accuracy steeply drops while the test accuracy plateaus.

They end up with accuracies of 0.5053 and 0.562 respectively so the network is just classifying everything the same. I do not understand how this behavior arises as I thought that balancing my training set would avoid the problem of classifying everything the same? Also, the training set begins to learn initially upwards from 50/50 but I cannot understand its reversion. Is there anything I can do to prevent this? Or should I just employ early stopping when the training accuracy begins to decrease?

Any insight would be appreciated!

opt = tf.keras.optimizers.SGD(lr=0.000001, momentum=0.9, decay=0, nesterov=True)

model = keras.Sequential([keras.layers.Dense(100,kernel_initializer='he_uniform',bias_initializer=keras.initializers.Constant(value=0.01),activation=tf.nn.relu,kernel_regularizer=regularizers.l2(0.1)), keras.layers.Dense(100,kernel_initializer='he_uniform',bias_initializer=keras.initializers.Constant(value=0.01),activation=tf.nn.relu, kernel_regularizer=regularizers.l2(0.1)), keras.layers.Dense(1, activation=tf.nn.sigmoid)])

model.compile(optimizer=opt, loss='binary_crossentropy', metrics=['accuracy']) enter image description here

  • $\begingroup$ It seems to me that the learning rate is too small... $\endgroup$
    – ignatius
    Jul 8, 2019 at 11:18
  • $\begingroup$ Thanks but unfortunately I've changed around the learning rate but it always ends up getting stuck on defaulting to the same loss. With a slower learning rate accuracy first goes up (as above) and then declines which I can't understand. $\endgroup$ Jul 8, 2019 at 12:13
  • $\begingroup$ I had problems with regularization in the past… I'd start the training without regularization $\endgroup$
    – ignatius
    Jul 8, 2019 at 12:21
  • $\begingroup$ I have tried that a little but I think I will explore regularization more yeah.....thanks for all the input!! $\endgroup$ Jul 8, 2019 at 12:23
  • $\begingroup$ 0.56 is not really imbalanced. You generally shouldn't have to do something to your data set above 0.9/0.1 imbalance. The problem here is that your model doesn't seem to learn. Maybe start by trying to overfit (big model, no regularisation). Edit: also I don't see it mentionned. How many instances do you have ? $\endgroup$ Jul 13, 2022 at 8:52

1 Answer 1


The fact tha balancing the dataset Will prevent the overfitting and thus, good results in the test set, Works under the assumption that the model does not underfit.

What it may be happening is that the model is too bad that it can't perform the classification task in either the un-balanced and the balanced dataset.

It's difficult to analyse the problema without more information, you may have facing convergence issues (too large learning rate, vanishing gradients), you may also be computing the accuracy in a bad way...

  • $\begingroup$ Thanks! I've tried changing the learning rate but to no avail and I don't think it is due to vanishing gradients because the network is shallow and I'm using RelU. With regards to accuracy, I am just using keras's accuracy metric. My loss is binary_crossentropy but this goes down to a certain value and then stops decreasing as the accuracy plateaus. $\endgroup$ Jul 8, 2019 at 10:06
  • $\begingroup$ I suggest you to post your network structure and training details $\endgroup$
    – ignatius
    Jul 8, 2019 at 10:17
  • $\begingroup$ opt = tf.keras.optimizers.SGD(lr=0.000001, momentum=0.9, decay=0, nesterov=True) model = keras.Sequential([keras.layers.Dense(100, kernel_initializer='he_uniform', bias_initializer=keras.initializers.Constant(value=0.01), activation=tf.nn.relu, kernel_regularizer=regularizers.l2(0.1)), keras.layers.Dense(100, kernel_initializer='he_uniform', bias_initializer=keras.initializers.Constant(value=0.01), activation=tf.nn.relu, kernel_regularizer=regularizers.l2(0.1)), keras.layers.Dense(1, activation=tf.nn.sigmoid)]) model.compile(optimizer=opt, loss='binary_crossentropy', metrics=['accuracy']) $\endgroup$ Jul 8, 2019 at 10:37
  • 1
    $\begingroup$ It's worth doing so on the original post, you can edit it $\endgroup$
    – ignatius
    Jul 8, 2019 at 10:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.