My task is to estimate a person's age based on a rgb image of the face of that person. I'm using ResNet-50 to that end. At first stage I trained my net on a dataset which is called WIKI-IMDB (after filtering and expanding my dataset by horizontal flipping it contains ~300k images) and I initialized the weights of the network with the weights of a pre-trained ResNet-50 network which trained on ImageNet. When training on the WIKI-IMDB dataset I calculated the mean of each channel (RGB) of the train set input and subtracted it from each example in the batch (I also used the same mean to subtract it from the validation set which I'm using for early stopping). after I finished training my network on WIKI-IMDB I want to test it on another dataset set which is a benchmark for this task. my new dataset which I'm testing on is called CACD and contains ~160k images which I'm dividing to train set and test set. I need to fine-tune my network on the CACD train set and finally report the metric I'm using on the CACD test set.
My question is: when learning on the new CACD train set, should I subtract the mean of the CACD train set from each example in the batch or should I keep subtracting the mean of the WIKI-IMDB train set? and if I need to subtract the new mean of the CACD train set, should I use that mean also when evaluating on the CACD test set?