1
$\begingroup$

I am doing transfer learning by retraining the publicly available inception layer, without regularisation here are my initial parameters and results:

training steps: 20,000
learning rate: 0.075
test accuracy :72.9%
train accuracy for final iteration : 98.0%
val accuracy for final iteration : 75.0%

clearly overfitting, so I tried L2 regularization. Here are my parameters and results, for the highest accuracy until now :

training steps: 40,000
learning rate: 0.1
test accuracy :71.1%
alpha for L2 : 0.00075
train accuracy for final iteration : 91.0%
val accuracy for final iteration : 71.0%

Looked promising so I decided to go further with the following hoping to get better accuracy :

training steps: 80,000
learning rate: 0.075
alpha for L2 : 0.005

go the following graph :

tensorboard graph

Since, the graph was getting saturated so I quit at 50,000 steps.

My question :

Should I continue with it, hoping it will get better or should I try with other values(Please suggest some good range of parameters that I should try)? or may be other techniques like L1 regularization or dropout ?

Note : Since, it is real-world data so getting new data would be difficult, and I cant flip the image to generate new data since orientation is also a factor to determine the label for output classes

$\endgroup$
  • 1
    $\begingroup$ Use drop out, it better performance than L regularizations. $\endgroup$ – Media May 6 '18 at 11:01
  • 1
    $\begingroup$ What batch size are you using? If it is small (e.g. less than 20), this would cause the large variance errors over iterations. $\endgroup$ – n1k31t4 Aug 4 '18 at 15:31
1
$\begingroup$

Steps for overcoming overfitting are the followings:

 1. Add more data  
 2. Use data augmentation  
 3. Use architectures that generalize well  
 4. Add regularization  
 5. Reduce architecture complexity.

I would try to add Dropout, lately it is being the most used regularization technique in complex networks architectures. Start adding it in the later layers, since it is best no to lose initial info. Something like 0.6 or like that, and if it is not enough add dropout in the previous layer, and so on.

$\endgroup$
  • $\begingroup$ currently I am just adding a single dense dense layer and feeding it to a softmax, should I add more dense layers? $\endgroup$ – Pratik Kumar Apr 6 '18 at 20:23
  • $\begingroup$ I am a bit lost, is it a CNN? can you post your network architecture? $\endgroup$ – Kailegh Apr 7 '18 at 18:10
  • $\begingroup$ yes, it is a CNN(look at my heading), architecture is inception_v3 whose softmax is replaced with new softmax of 7 categories. $\endgroup$ – Pratik Kumar Apr 7 '18 at 19:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.