I understand that one should standardize and normalize the test data (or any "unlabeled" data) with the training mean and sd. How can I implement this in R language? Is there a kind of "fitting" to the training set and a kind of applying to the test data?


Check out the preProcess function from the caret library. You can choose the parameters you want to scale/center the training data, and it also saves the transformations it makes so then you can normalize the test set with the same specifications that you normalized the training set with. Could go something like this:

trainData <- data.frame(v1 = rnorm(15,3,1), v2 = rnorm(15,2,2))
testData <- data.frame(v1 = rnorm(5,3,1), v2 = rnorm(5,2,2))
normParam <- preProcess(trainData)
norm.testData <- predict(normParam, testData)

now your norm.testData is scaled and centered according to the training data set parameters.

Another way to do this without using caret:

## set up data
trainData <- data.frame(v1 = rnorm(15,3,1), v2 = rnorm(15,2,2))
testData <- data.frame(v1 = rnorm(5,3,1), v2 = rnorm(5,2,2))

## find mean and sd column-wise of training data
trainMean <- apply(trainData,2,mean)
trainSd <- apply(trainData,2,sd)

## centered
sweep(trainData, 2L, trainMean) # using the default "-" to subtract mean column-wise   
## centered AND scaled
norm2.testData <- sweep(sweep(testData, 2L, trainMean), 2, trainSd, "/")
  • $\begingroup$ I must omit the label column from the training data to leave it at original scale, right? (Weird that this step mostly stays unnoticed). $\endgroup$ – Hendrik Sep 13 '16 at 14:35
  • $\begingroup$ Correct. The labels should remain unchanged when normalizing. $\endgroup$ – TBSRounder Sep 13 '16 at 14:50
  • $\begingroup$ Thanks. While I accept the reasoning to take the training set into account as a basis for standardization in general, I wonder if including the otherwise available test set (in case of a competition, for instance) into the process would be better, perhaps. I would combine both the train (minus label) and data sets and "fit" preProcess to this combined sets. Isn't it better? $\endgroup$ – Hendrik Sep 13 '16 at 15:16
  • 1
    $\begingroup$ You would probably get a "better" performance, but only because you are overfitting. You would be using the "gold standard" test set to influence how your training data is pre-processed, which is not good, and when you try predicting things past your now-tainted test set, you will probably see a performance drop because of that overfitting. Every case/data are different, but that is the general idea of why thats not a good way to do it. $\endgroup$ – TBSRounder Sep 13 '16 at 15:43
  • $\begingroup$ As for using Caret: apart from the fact that I try to evade the overdependence from specific packages and like to learn mechanisms "by hand", Caret cannot provide all possibly necessary imputation method (in my present case, "mean imputation"). So it would still be handy to have a more general solution to implement any kind of (columnwise!) standardization/normalization. $\endgroup$ – Hendrik Sep 14 '16 at 7:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.