# Feature Scaling and normalization in cross-validation set

I have a question that normally, when we are making a training set and a final test set,

we would compute the mean and standard deviation for preprocessing using the training data and use it to standardize (transform) the test data.

So, when we are making a training data, a cross-validation data and a final test data, shouldn't we compute mean and std for preprocessing using the training data alone and then standardize the validation set and test set using it? And, if I'm right how can we do it in keras, since in keras like in the below code, taken from kaggle( by Francois Chollet MNIST tutorial):

simple-deep-mlp-with-keras by Francois Chollet

# pre-processing: divide by max and substract mean
scale = np.max(X_train)
X_train /= scale
X_test /= scale

mean = np.std(X_train)
X_train -= mean
X_test -= mean

model.fit(X_train, y_train, nb_epoch=10, batch_size=16, validation_split=0.1, show_accuracy=True, verbose=2)


The validation set is created using the fit method and thus, it is also used in computing the preprocessing mean and std. So, how to create validation set before and use only the training data for feature scaling and normalization? And which method is the right method?

Also, in the above code scale and mean are being computed for each column respectively and is being used for normalization , should we compute mean and scale(standard deviation or range) for each pixel or for the entire image? How does that affect?

Answering your last question first: your mean and std should be computed over the entire image. So for an RGB image you will get a mean/std vector of 3 values for the entire dataset (calculated per channel). Normalizing per pixel is wrong and would result in loss of information (you need to think of an image as a single discreet function, you can't treat each part of it separately without changing its overall meaning).

Regarding normalizing the validation set, you can manually create the validation set and use the "validation_data" argument in the fit method:

# Creating the validation set:
X_valid = X_train[int(0.9*len(X_train)):]
X_train = X_train[:int(0.9*len(X_train))]
y_valid = y_train[int(0.9*len(y_train)):]
y_train = y_train[:int(0.9*len(y_train))]

# pre-processing: divide by max and subtract mean

scale = np.max(X_train)
X_train /= scale
X_valid /= scale
X_test /= scale

mean = np.std(X_train)
X_train -= mean
X_valid -= mean
X_test -= mean

model.fit(X_train, y_train, batch_size=16, epochs=10, validation_data=(X_valid, y_valid), verbose=2)