I'm still training a Convolutional Neural Network model in Tensorflow to recognize age groups from facial images. I'm using the Adience 3D (frontalized with a 3D model - it might not be good data) dataset, with four folds for training, and one fold for testing.
I'm currently using three Conv-Conv-Pool layer sets, the first one's Conv layer using 32 filters with 5x5 size, the second and third using 64 filters with 3x3 size, and to top it all off, a Dense layer with flattened input, then a Dropout layer, dropping out the outputs of the dense layer by 20%, then on to the logits layer.
I normalized my cropped color images with the per-channel normalization - the
(A - np.mean(A)) / np.std(A) method for each 64x64 array in the 64x64x3 input.
Currently I'm training it with Batch Size 64 and 100 steps so as to evaluate the weights every 100 steps, and while I don't see much improvement for now, I could see it increasing from steps 100 to 500 (0.27 to 0.35), before dropping on stage 600 then going up again up to step 800 (0.30 - 0.34).
Is this a sign I have to change my layer structure?
On that note, when should I change up my layer structure/hyperparameters like training steps, batch size, filter size, etc., and how?