I am doing transfer learning by retraining the publicly available inception layer, without regularisation here are my initial parameters and results:

training steps: 20,000
learning rate: 0.075
test accuracy :72.9%
train accuracy for final iteration : 98.0%
val accuracy for final iteration : 75.0%

clearly overfitting, so I tried L2 regularization. Here are my parameters and results, for the highest accuracy until now :

training steps: 40,000
learning rate: 0.1
test accuracy :71.1%
alpha for L2 : 0.00075
train accuracy for final iteration : 91.0%
val accuracy for final iteration : 71.0%

Looked promising so I decided to go further with the following hoping to get better accuracy :

training steps: 80,000
learning rate: 0.075
alpha for L2 : 0.005

go the following graph :

tensorboard graph

Since, the graph was getting saturated so I quit at 50,000 steps.

My question :

Should I continue with it, hoping it will get better or should I try with other values(Please suggest some good range of parameters that I should try)? or may be other techniques like L1 regularization or dropout ?

Note : Since, it is real-world data so getting new data would be difficult, and I cant flip the image to generate new data since orientation is also a factor to determine the label for output classes

  • 1
    $\begingroup$ Use drop out, it better performance than L regularizations. $\endgroup$ Commented May 6, 2018 at 11:01
  • 1
    $\begingroup$ What batch size are you using? If it is small (e.g. less than 20), this would cause the large variance errors over iterations. $\endgroup$
    – n1k31t4
    Commented Aug 4, 2018 at 15:31

1 Answer 1


Steps for overcoming overfitting are the followings:

 1. Add more data  
 2. Use data augmentation  
 3. Use architectures that generalize well  
 4. Add regularization  
 5. Reduce architecture complexity.

I would try to add Dropout, lately it is being the most used regularization technique in complex networks architectures. Start adding it in the later layers, since it is best no to lose initial info. Something like 0.6 or like that, and if it is not enough add dropout in the previous layer, and so on.

  • $\begingroup$ currently I am just adding a single dense dense layer and feeding it to a softmax, should I add more dense layers? $\endgroup$ Commented Apr 6, 2018 at 20:23
  • $\begingroup$ I am a bit lost, is it a CNN? can you post your network architecture? $\endgroup$
    – Kailegh
    Commented Apr 7, 2018 at 18:10
  • $\begingroup$ yes, it is a CNN(look at my heading), architecture is inception_v3 whose softmax is replaced with new softmax of 7 categories. $\endgroup$ Commented Apr 7, 2018 at 19:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.