Is it common that a model only learns in a few training runs and doesn't learn in the majority of training runs

Question

I have a model that does binary classification. I train the same model many times, say, 20 times. In these 20 training runs:

1. roughly 10% of the training attempts learn fast and the model performs well (e.g., reaching 95% AUC) in a few epochs.
2. roughly 80% of the training attempts do not learn at all (i.e., getting stuck at 50% AUC) after 100 epochs.
3. A few training runs are in between, meaning that they get stuck at 50% AUC for many epochs but after a while they learn as fast as the models in the first scenario.

My guess is that perhaps somehow the loss function has a very sharp downward spike like this:

So it needs some "luck" for the optimizer and initial weights to be "just right" to find it.

My questions are:

1. Is this (i.e., only a few training runs give us good result while the rest of them do not improve the model at all) common in the industry?

If the answer is yes, I can think of the following:

• Tune the learning rate (which is I am currently trying, I reduced the learning rate, so that hopefully the optimizer will not "miss" the trough)
• Try different optimizers, SGD/Adam/RMSprop
• Try some initialization techniques (Xavier Glorot? I didn't explore much in this regard yet)

2. Is there any other thing I can do to improve this?

EDIT1 For the sake of completeness of the question, a simplified version of the model is pasted here:

model = keras.Sequential([     
    layers.Conv2D(filters=64, kernel_size=(3, 3), activation='relu'),
    layers.Conv2D(filters=64, kernel_size=(3, 3), activation='relu'),
    layers.MaxPooling2D(pool_size=(2, 2),strides=(2, 2)),
    layers.Flatten(),

    layers.Dense(4096, activation='relu'),
    layers.Dense(1, activation='sigmoid')
])


model.build((None,) + input_shape)

optimizer = keras.optimizers.Adam(learning_rate=5e-5)
#optimizer = keras.optimizers.SGD(learning_rate=1.0e-3, nesterov=True)
model.compile(
    optimizer=optimizer,
    loss=tf.keras.losses.BinaryCrossentropy(),
    metrics=[tf.keras.metrics.AUC() , tf.keras.metrics.AUC(curve='PR')]
)

Answer 1

You are probably right that such inconstant behavior indicates that you result greatly depends on how lucky you are with the starting random weights.

You probably need to find the right number for learning rate and the number of epochs.

You can try a learning rate finder in order to find the optimal learning rate automatically.

Also, you did not specify what optimizer you are currently using, but I would suggest you try binary cross entropy since you are doing a binary classification.