My task is to estimate a person's age based on a face image of that person.
To that end I'm using a CNN and at first stage I was based on the following article: DeepExpectation which uses a VGG16 architecture to predict a person apparent age (the age that other people would vote). I'm using ResNet 50 architecture (and I'm using this implementation of it: ResNet Tensorflow in tensorflow). The dataset I use for learning is taken from the same article above and can be downloaded from here: WIKI-IMDB dataset. It is composed of 523,051 face images with tagged ages but I found that most of them are garbage (doesn't have real age as the label or has more than one face in it or no face at all). after filtering this dataset (by throwing images that have more than one face or no face at all or the label that represents the age doesn't make sense) I'm left with approximately 150k images in my dataset. in the process of making this dataset I centered all faces in the image and cropped it to be 224*224*3. To increase the size of my dataset I flipped horizontally each image in my dataset to get a total of 300k images in my dataset. I then split the dataset to 240k train, 30k validation and 30k test images.

I'm loading a pre-trained model (trained on ImageNet) and I hoped to get the results they got in the article above (something close to 3.2 years with MAE as the evaluation metric). I also should mention that I'm using cross-entropy as the loss and an l2 loss for regularization and I have 101 classes as the logits (ages 0-100). The batch size I'm using is 64 (The maximum my GPU can handle with). I tried several combinations of hyperparameters but until now the best MAE I got on the validation set was 5.8 years.

Examples of loss graphs I got and their hyperparameters:
weight decay = 5e-4,
momentum optimizer with momentum=0.9,
first learning rate = 0.01 and reducing it by a factor of 10 each 2 epochs until reaching to 1e-6,
the above LR is the LR assinged to the last FC layer. for the layer before it I used 0.5 of that LR. for the middle layers I used 1e-2 of that LR and for the first layer I used 1e-3 of that LR.
I got the following graphs: (orange = train , blue = validation)
loss: Loss
LR: enter image description here
MAE: enter image description here

example #2:
same hyperparameters as in the example above only different ratios between first layers and last FC layer. in this example I used 0.5 of the LR in the last FC layer for the layer before it and 0.1 of the LR in the last FC layer for the other layers.
I got the following graphs: (orange = train , blue = validation)
loss: enter image description here
LR: enter image description here
MAE: enter image description here

As you can see from the last 2 examples, I'm reaching to ~6 pretty fast but it seems as the optimization on the train loss gets stuck and also that the validation loss doesn't approach the train loss from some point. Do you have any suggestions?

Also, I'm having some thoughts on whether it is good to apply some of the following:

  1. Initalize weights from the pre-trained network until some layer
  2. Initialize all weights from the pre-trained network beside the batch-norm
  3. Freeze some of the layers after loading them and train only the last layers
  4. Apply different LR to different layers (as I did in the examples above)
  5. Should I use Momentum optimizer or maybe Adam Optimizer
  6. If I'm using Momentum optimizer, what should be the learning rate schedule (learning rate decay)?
  7. If I'm using Adam optimizer, does it make sense to use different learning rates to different layers?
  8. What should be the weight decay hyperparameter?

Any help would be much appreciated.

I didn't know which stackexchange site is more appropriate for this question so I also posted this question on CrossValidated: Difficulty in choosing Hyperparameters for my CNN


1 Answer 1


I am going to comment on your ideas:

  1. This is generally recommended. There is no proven reason not to do it but many reasons to do it, and you should initialize as much weights as possible. Typically it is done until the last layer.
  2. Initialize everything that you can from pre-trained network.
  3. I would freeze the first layers in the first epochs, and after that unfreeze them. If possible, I would rather use differential learning rates, as this post shows.
  4. Totally, as shown above.
  5. Adam is more widely used, I would use that.
  6. Cosine scheduling with reestarts has been shown to be very effective, but any learning rate with reestarts should be similar.
  7. It does make sense.
  8. I wouldn't worry about that until you are overfitting, and you are not overfitting yet. If you overfit, I would recommend dropout instead of weight decay.
  • $\begingroup$ thanks for your answer. with regard to points 3 and 8: If I use your suggested method and freeze the first layers in the first few epochs and then unfreeze them, does it go well while combining it with different LR's? and regarding point 8: for my understanding, the reason for not using dropout in ResNet architecture is that the BN layers sort of replacing them as regularizers. I'm not quite sure why they put the weight decay in addition to the BN layers but I'm not sure also that applying dropout will be a good combination with the BN layers. $\endgroup$
    – Vegetal605
    Commented May 28, 2018 at 8:15
  • $\begingroup$ If the BN layers already act as regularizers, you can try not to use any regularization technique. If you overfit, you can use dropout to regularize more (bn layers regularize a little, but not as much as dropout). About point 3: the best idea is using differential learning rates. If you don't know how to implement that, you can freeze layers instead. $\endgroup$ Commented May 28, 2018 at 8:24
  • $\begingroup$ As written in my question, I'm already using differential learning rates but I understood your point. $\endgroup$
    – Vegetal605
    Commented May 28, 2018 at 8:35
  • $\begingroup$ Did I answer your question? $\endgroup$ Commented May 29, 2018 at 8:22
  • $\begingroup$ Your answer did help me. Especially the Cosine scheduling with restarts. It did the job with regards to the optimization on the train set but now I run into overfit at some point and I at first I tried just to enlarge the weight decay parameter but that didn't help much (only slightly). I wanted to take your advice and put in some dropout layers but I'm not quite sure where should I put them. in some papers 2 years ago they usually put the dropout after FC layers. but in ResNet architecture there is only 1 FC layer (the last layer)...so where should I put the dropout layers? $\endgroup$
    – Vegetal605
    Commented May 29, 2018 at 13:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.