I have a model that does binary classification. I train the same model many times, say, 20 times. In these 20 training runs:

  1. roughly 10% of the training attempts learn fast and the model performs well (e.g., reaching 95% AUC) in a few epochs.
  2. roughly 80% of the training attempts do not learn at all (i.e., getting stuck at 50% AUC) after 100 epochs.
  3. A few training runs are in between, meaning that they get stuck at 50% AUC for many epochs but after a while they learn as fast as the models in the first scenario.

My guess is that perhaps somehow the loss function has a very sharp downward spike like this: the loss function with a very sharp downward spike

So it needs some "luck" for the optimizer and initial weights to be "just right" to find it.

My questions are:

1. Is this (i.e., only a few training runs give us good result while the rest of them do not improve the model at all) common in the industry?

If the answer is yes, I can think of the following:

  • Tune the learning rate (which is I am currently trying, I reduced the learning rate, so that hopefully the optimizer will not "miss" the trough)
  • Try different optimizers, SGD/Adam/RMSprop
  • Try some initialization techniques (Xavier Glorot? I didn't explore much in this regard yet)

2. Is there any other thing I can do to improve this?

EDIT1 For the sake of completeness of the question, a simplified version of the model is pasted here:

model = keras.Sequential([     
    layers.Conv2D(filters=64, kernel_size=(3, 3), activation='relu'),
    layers.Conv2D(filters=64, kernel_size=(3, 3), activation='relu'),
    layers.MaxPooling2D(pool_size=(2, 2),strides=(2, 2)),

    layers.Dense(4096, activation='relu'),
    layers.Dense(1, activation='sigmoid')

model.build((None,) + input_shape)

optimizer = keras.optimizers.Adam(learning_rate=5e-5)
#optimizer = keras.optimizers.SGD(learning_rate=1.0e-3, nesterov=True)
    metrics=[tf.keras.metrics.AUC() , tf.keras.metrics.AUC(curve='PR')]

1 Answer 1


You are probably right that such inconstant behavior indicates that you result greatly depends on how lucky you are with the starting random weights.

You probably need to find the right number for learning rate and the number of epochs.

You can try a learning rate finder in order to find the optimal learning rate automatically.

Also, you did not specify what optimizer you are currently using, but I would suggest you try binary cross entropy since you are doing a binary classification.

  • 1
    $\begingroup$ Hi @Silver Light, I added a simplified version of my model to the question. On learning rate finder, I will take a closer look. On the loss function, I am using binary cross-entropy already. $\endgroup$ Commented May 23, 2023 at 11:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.