I'm training a segmentation model, Unet++, on 2d images and I am now trying to find the optimal learning rate. The backbone of the model is Resnet34, I use Adam optimizer and the loss function is the dice loss function.
Also, I use a few callbacksfunctions:
callbacks = [ keras.callbacks.EarlyStopping(monitor='val_loss', patience=15, verbose=1, min_delta=epsilon, mode='min'), keras.callbacks.ReduceLROnPlateau(monitor='val_loss', factor=0.5, patience=3, verbose=1, mode='min', cooldown=0, min_lr=1e-8), keras.callbacks.ModelCheckpoint(model_save_path, save_weights_only=True, save_best_only=True, mode='min'), keras.callbacks.ReduceLROnPlateau(), keras.callbacks.CSVLogger(logger_save_path) ]
I plotted the curves of training loss over epochs for a few learning rates:
The validation loss and training loss seem to decrease slowly. However, the validation loss isn't oscillating (it is almost always decreasing).
The validation and training losses decreased quickly on first 2/3 epochs. After 6 or 7 epochs, the validation loss increases again.
I have a few questions (I hope it is not too much):
- *What is normally the best way to find the learning rate i.e.
- How many epochs should I wait before considering that the learning rate isn't good?
- What are the criteria on the loss function to determine if a learning rate is "good"?
- Is there a big difference if I use a small learning (which still converges) instead of the "optimal" learning rate ?
- Is it normal that the validation loss function oscillates over the training?
- Which learning rate should I use according to my results?
Even a partial response would help me a lot.