I understand that one should standardize and normalize the test data (or any "unlabeled" data) with the training mean and sd. How can I implement this in R language? Is there a kind of "fitting" to the training set and a kind of applying to the test data?
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
7
Check out the preProcess function from the caret library. You can choose the parameters you want to scale/center the training data, and it also saves the transformations it makes so then you can normalize the test set with the same specifications that you normalized the training set with. Could go something like this:
library(caret)
trainData <- data.frame(v1 = rnorm(15,3,1), v2 = rnorm(15,2,2))
testData <- data.frame(v1 = rnorm(5,3,1), v2 = rnorm(5,2,2))
normParam <- preProcess(trainData)
norm.testData <- predict(normParam, testData)
now your norm.testData is scaled and centered according to the training data set parameters.
Another way to do this without using caret:
## set up data
trainData <- data.frame(v1 = rnorm(15,3,1), v2 = rnorm(15,2,2))
testData <- data.frame(v1 = rnorm(5,3,1), v2 = rnorm(5,2,2))
## find mean and sd column-wise of training data
trainMean <- apply(trainData,2,mean)
trainSd <- apply(trainData,2,sd)
## centered
sweep(trainData, 2L, trainMean) # using the default "-" to subtract mean column-wise
## centered AND scaled
norm2.testData <- sweep(sweep(testData, 2L, trainMean), 2, trainSd, "/")
-
$\begingroup$ I must omit the label column from the training data to leave it at original scale, right? (Weird that this step mostly stays unnoticed). $\endgroup$– HendrikSep 13, 2016 at 14:35
-
$\begingroup$ Correct. The labels should remain unchanged when normalizing. $\endgroup$ Sep 13, 2016 at 14:50
-
$\begingroup$ Thanks. While I accept the reasoning to take the training set into account as a basis for standardization in general, I wonder if including the otherwise available test set (in case of a competition, for instance) into the process would be better, perhaps. I would combine both the train (minus label) and data sets and "fit" preProcess to this combined sets. Isn't it better? $\endgroup$– HendrikSep 13, 2016 at 15:16
-
1$\begingroup$ You would probably get a "better" performance, but only because you are overfitting. You would be using the "gold standard" test set to influence how your training data is pre-processed, which is not good, and when you try predicting things past your now-tainted test set, you will probably see a performance drop because of that overfitting. Every case/data are different, but that is the general idea of why thats not a good way to do it. $\endgroup$ Sep 13, 2016 at 15:43
-
$\begingroup$ As for using Caret: apart from the fact that I try to evade the overdependence from specific packages and like to learn mechanisms "by hand", Caret cannot provide all possibly necessary imputation method (in my present case, "mean imputation"). So it would still be handy to have a more general solution to implement any kind of (columnwise!) standardization/normalization. $\endgroup$– HendrikSep 14, 2016 at 7:46