Difficulty in choosing Hyperparameters for my CNN

Question

My task is to estimate a person's age based on a face image of that person.
To that end I'm using a CNN and at first stage I was based on the following article: DeepExpectation which uses a VGG16 architecture to predict a person apparent age (the age that other people would vote). I'm using ResNet 50 architecture (and I'm using this implementation of it: ResNet Tensorflow in tensorflow). The dataset I use for learning is taken from the same article above and can be downloaded from here: WIKI-IMDB dataset. It is composed of 523,051 face images with tagged ages but I found that most of them are garbage (doesn't have real age as the label or has more than one face in it or no face at all). after filtering this dataset (by throwing images that have more than one face or no face at all or the label that represents the age doesn't make sense) I'm left with approximately 150k images in my dataset. in the process of making this dataset I centered all faces in the image and cropped it to be 224*224*3. To increase the size of my dataset I flipped horizontally each image in my dataset to get a total of 300k images in my dataset. I then split the dataset to 240k train, 30k validation and 30k test images.

I'm loading a pre-trained model (trained on ImageNet) and I hoped to get the results they got in the article above (something close to 3.2 years with MAE as the evaluation metric). I also should mention that I'm using cross-entropy as the loss and an l2 loss for regularization and I have 101 classes as the logits (ages 0-100). The batch size I'm using is 64 (The maximum my GPU can handle with). I tried several combinations of hyperparameters but until now the best MAE I got on the validation set was 5.8 years.

Examples of loss graphs I got and their hyperparameters:
weight decay = 5e-4,
momentum optimizer with momentum=0.9,
first learning rate = 0.01 and reducing it by a factor of 10 each 2 epochs until reaching to 1e-6,
the above LR is the LR assinged to the last FC layer. for the layer before it I used 0.5 of that LR. for the middle layers I used 1e-2 of that LR and for the first layer I used 1e-3 of that LR.
I got the following graphs: (orange = train , blue = validation)
loss:
LR:
MAE:

example #2:
same hyperparameters as in the example above only different ratios between first layers and last FC layer. in this example I used 0.5 of the LR in the last FC layer for the layer before it and 0.1 of the LR in the last FC layer for the other layers.
I got the following graphs: (orange = train , blue = validation)
loss:
LR:
MAE:

As you can see from the last 2 examples, I'm reaching to ~6 pretty fast but it seems as the optimization on the train loss gets stuck and also that the validation loss doesn't approach the train loss from some point. Do you have any suggestions?

Also, I'm having some thoughts on whether it is good to apply some of the following:

1. Initalize weights from the pre-trained network until some layer
2. Initialize all weights from the pre-trained network beside the batch-norm
3. Freeze some of the layers after loading them and train only the last layers
4. Apply different LR to different layers (as I did in the examples above)
5. Should I use Momentum optimizer or maybe Adam Optimizer
6. If I'm using Momentum optimizer, what should be the learning rate schedule (learning rate decay)?
7. If I'm using Adam optimizer, does it make sense to use different learning rates to different layers?
8. What should be the weight decay hyperparameter?

Any help would be much appreciated.

I didn't know which stackexchange site is more appropriate for this question so I also posted this question on CrossValidated: Difficulty in choosing Hyperparameters for my CNN

Answer 1

I am going to comment on your ideas:

1. This is generally recommended. There is no proven reason not to do it but many reasons to do it, and you should initialize as much weights as possible. Typically it is done until the last layer.
2. Initialize everything that you can from pre-trained network.
3. I would freeze the first layers in the first epochs, and after that unfreeze them. If possible, I would rather use differential learning rates, as this post shows.
4. Totally, as shown above.
5. Adam is more widely used, I would use that.
6. Cosine scheduling with reestarts has been shown to be very effective, but any learning rate with reestarts should be similar.
7. It does make sense.
8. I wouldn't worry about that until you are overfitting, and you are not overfitting yet. If you overfit, I would recommend dropout instead of weight decay.