I'm using caffenet for fine-tuning. I'm doing cross validation (15 vs all) with a very small data set of about 250 images. I'm testing every 10 iterations (~2 epochs). My batch size is 50. With some sets I'm getting very unstable accuracy - Can jump from 70% to 90% and back to 70% and back and fourth. My question is: Let's say I hit 90% accuracy after 40 iterations (~8 epochs) - Does this mean that the net had reached an optimal state or could it be that it just had a lucky guess on the validation set? My final question is: Should I stop training and save the net? Thanks.
2
-
$\begingroup$ 250 images is a very small training set. I'm not surprised that you would have all sorts of problems training a CNN on such a small dataset. $\endgroup$– D.W.Apr 30, 2017 at 5:50
-
$\begingroup$ @D.W. You'll be glad to hear that I'm actually getting very good results with 190 images from 2 classes. To be fair - It's probably a more simple classification task than cats-vs-dogs or something.. It's just that the third class is a bit confusing to the net.. $\endgroup$– Gil-MorApr 30, 2017 at 19:14
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
5
If accuracy regresses something is wrong in either the network, or (more likely here) the meta-parameters (probably learning rate.)
It can be difficult to tell when a model converges. I'd recommend looking at diagnostic graphs (typically training loss, training/validation accuracy, and ratio of weights:updates) over epochs. Typically convergence is considered when loss and accuracy level out and show diminishing returns beyond some threshold (your tolerance for 1.0e-x% improvements.) So, stop training/validation when it's improving less than what you care about.
-
$\begingroup$ Thanks, I'm almost certain that my data is what makes it hard for the net to converge. Some of the images of one class might look like images from the two other classes. But anyway, You say that if my net doesn't converge than It doesn't learn even if it reaches 90% accuracy in some cycles.. $\endgroup$– Gil-MorApr 29, 2017 at 10:05
-
$\begingroup$ If the network does not pick up on features which make images different, it can certainly cause problems. "You say that if my net doesn't converge than It doesn't learn even if it reaches 90% accuracy in some cycles." Converge just means it has reached its optimal solution (when the training loss and training/validation accuracy level out.) 90% accuracy in a training epoch is good (and may be convergence for the architecture/data combination), but you then need the validation and test accuracy as well. Typically it is lower for each (training > validation > test accuracy.) $\endgroup$– KadeGApr 30, 2017 at 23:40
-
$\begingroup$ Ran out of room... As other have said though, you're working with a tiny dataset. Consider getting more data. $\endgroup$– KadeGApr 30, 2017 at 23:42
-
$\begingroup$ Thanks @Kade, I get 90% accuracy in validation cycle on the validation set.. Also, regarding the dataset size - I answered to a comment by D.W - " You'll be glad to hear that I'm actually getting very good results with 190 images from 2 classes. To be fair - It's probably a more simple classification task than cats-vs-dogs or something.. It's just that the third class is a bit confusing to the net." $\endgroup$– Gil-MorMay 1, 2017 at 9:04
-
$\begingroup$ but anyway I believe that the net isn't learning the dataset with 3 classes and that the 90% accuracy in some ephocs is just lucky guesses on the 15 images validation set. $\endgroup$– Gil-MorMay 1, 2017 at 9:07