1
$\begingroup$

So I am training a neural network on a binary classification problem and my Case (1) and Controls (0) were imbalanced so I oversampled my cases so that that the training set was 0.5053 made up of controls. I did not balance my test set which was 0.562 controls. In the beginning my train and test accuracy rises (it is not very accurate still but I expect this to be the case) but then the training accuracy steeply drops while the test accuracy plateaus.

They end up with accuracies of 0.5053 and 0.562 respectively so the network is just classifying everything the same. I do not understand how this behavior arises as I thought that balancing my training set would avoid the problem of classifying everything the same? Also, the training set begins to learn initially upwards from 50/50 but I cannot understand its reversion. Is there anything I can do to prevent this? Or should I just employ early stopping when the training accuracy begins to decrease?

Any insight would be appreciated!

opt = tf.keras.optimizers.SGD(lr=0.000001, momentum=0.9, decay=0, nesterov=True)

model = keras.Sequential([keras.layers.Dense(100,kernel_initializer='he_uniform',bias_initializer=keras.initializers.Constant(value=0.01),activation=tf.nn.relu,kernel_regularizer=regularizers.l2(0.1)), keras.layers.Dense(100,kernel_initializer='he_uniform',bias_initializer=keras.initializers.Constant(value=0.01),activation=tf.nn.relu, kernel_regularizer=regularizers.l2(0.1)), keras.layers.Dense(1, activation=tf.nn.sigmoid)])

model.compile(optimizer=opt, loss='binary_crossentropy', metrics=['accuracy']) enter image description here

$\endgroup$
  • $\begingroup$ It seems to me that the learning rate is too small... $\endgroup$ – ignatius Jul 8 '19 at 11:18
  • $\begingroup$ Thanks but unfortunately I've changed around the learning rate but it always ends up getting stuck on defaulting to the same loss. With a slower learning rate accuracy first goes up (as above) and then declines which I can't understand. $\endgroup$ – Ciaran Kelly Jul 8 '19 at 12:13
  • $\begingroup$ I had problems with regularization in the past… I'd start the training without regularization $\endgroup$ – ignatius Jul 8 '19 at 12:21
  • $\begingroup$ I have tried that a little but I think I will explore regularization more yeah.....thanks for all the input!! $\endgroup$ – Ciaran Kelly Jul 8 '19 at 12:23
1
$\begingroup$

The fact tha balancing the dataset Will prevent the overfitting and thus, good results in the test set, Works under the assumption that the model does not underfit.

What it may be happening is that the model is too bad that it can't perform the classification task in either the un-balanced and the balanced dataset.

It's difficult to analyse the problema without more information, you may have facing convergence issues (too large learning rate, vanishing gradients), you may also be computing the accuracy in a bad way...

$\endgroup$
  • $\begingroup$ Thanks! I've tried changing the learning rate but to no avail and I don't think it is due to vanishing gradients because the network is shallow and I'm using RelU. With regards to accuracy, I am just using keras's accuracy metric. My loss is binary_crossentropy but this goes down to a certain value and then stops decreasing as the accuracy plateaus. $\endgroup$ – Ciaran Kelly Jul 8 '19 at 10:06
  • $\begingroup$ I suggest you to post your network structure and training details $\endgroup$ – ignatius Jul 8 '19 at 10:17
  • $\begingroup$ opt = tf.keras.optimizers.SGD(lr=0.000001, momentum=0.9, decay=0, nesterov=True) model = keras.Sequential([keras.layers.Dense(100, kernel_initializer='he_uniform', bias_initializer=keras.initializers.Constant(value=0.01), activation=tf.nn.relu, kernel_regularizer=regularizers.l2(0.1)), keras.layers.Dense(100, kernel_initializer='he_uniform', bias_initializer=keras.initializers.Constant(value=0.01), activation=tf.nn.relu, kernel_regularizer=regularizers.l2(0.1)), keras.layers.Dense(1, activation=tf.nn.sigmoid)]) model.compile(optimizer=opt, loss='binary_crossentropy', metrics=['accuracy']) $\endgroup$ – Ciaran Kelly Jul 8 '19 at 10:37
  • 1
    $\begingroup$ It's worth doing so on the original post, you can edit it $\endgroup$ – ignatius Jul 8 '19 at 10:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.