I have a model that does binary classification. I train the same model many times, say, 20 times. In these 20 training runs:
- roughly 10% of the training attempts learn fast and the model performs well (e.g., reaching 95% AUC) in a few epochs.
- roughly 80% of the training attempts do not learn at all (i.e., getting stuck at 50% AUC) after 100 epochs.
- A few training runs are in between, meaning that they get stuck at 50% AUC for many epochs but after a while they learn as fast as the models in the first scenario.
So it needs some "luck" for the optimizer and initial weights to be "just right" to find it.
My questions are:
1. Is this (i.e., only a few training runs give us good result while the rest of them do not improve the model at all) common in the industry?
If the answer is yes, I can think of the following:
- Tune the learning rate (which is I am currently trying, I reduced the learning rate, so that hopefully the optimizer will not "miss" the trough)
- Try different optimizers, SGD/Adam/RMSprop
- Try some initialization techniques (Xavier Glorot? I didn't explore much in this regard yet)
2. Is there any other thing I can do to improve this?
EDIT1 For the sake of completeness of the question, a simplified version of the model is pasted here:
model = keras.Sequential([ layers.Conv2D(filters=64, kernel_size=(3, 3), activation='relu'), layers.Conv2D(filters=64, kernel_size=(3, 3), activation='relu'), layers.MaxPooling2D(pool_size=(2, 2),strides=(2, 2)), layers.Flatten(), layers.Dense(4096, activation='relu'), layers.Dense(1, activation='sigmoid') ]) model.build((None,) + input_shape) optimizer = keras.optimizers.Adam(learning_rate=5e-5) #optimizer = keras.optimizers.SGD(learning_rate=1.0e-3, nesterov=True) model.compile( optimizer=optimizer, loss=tf.keras.losses.BinaryCrossentropy(), metrics=[tf.keras.metrics.AUC() , tf.keras.metrics.AUC(curve='PR')] )