Preprocessing to-be-predicted data in ML with R - "learn" and "apply" features

Question

I have studied the usual preprocessing methods for Machine Learning but I couldn't cope the following specific problem.

I apply the "usual" preparation for modeling (dummy variables, normalization, PCA etc., of course in the necessary cases) to the training data. So far, so good. But when I get the to-be-classified new data to make prediction the model constructed above, it's evident that I must apply these preparatory steps to this new data set as well.

And the problem arises here, because if I simply apply the preparatory steps for my new data in turn again, these doesn't take into consideration the characteristics of the training data. So, if I convert the new data factors into dummies, then it takes only the existing factor levels in the new data into account; if I min-max normalize the new data, it will be normalized according its own min-max values, disregarding the values in the training data; and if I use PCA, then the resulting components from the new data will be totally independent of the training data.

So essentially my point is that applying the same conversions separately to the train set and the new data set(s) (which could be only one observation as well), then the two resulting transformed sets will have nothing in common, so the prediction will be baseless.

I found some traces that in some cases there is some "learning" step in the training phase in these transformations as well and apply this "knowledge" to new data (caret and Sklearn, for instance, with "predict" could transform to the new data with characteristics learned from the training data), but generally speaking this inconsistency remains unmentioned otherwise.

What is the correct practice here?

Answer 1

In short, the correct practice is to use the scaling/preprocessing parameters you found and used on your training set, and apply them to your test set. the motivation for this is that you should be able to deal with someone coming to you with one new observation they want a prediction on. You will be able to preprocess that single observation using your training set preprocessing parameters.

Ill go through some of the issues you brought up:

Dummy Variables: This might entail you using setdiff() in R to find what columns are in your training set but not in your test set, then make a column in your test set with a column of zeros. This is the simplest way to do it, but it is very data/purpose dependent. You may also want to change this or look for better options depending on what modeling technique you're using.

Min/Max Normalize: When you scale your training set, save the min and max of each column, and scale your test set using those parameters (NOT the min/max of your test set).

PCA: When you perform PCA on your training set, you are transform your data based on rotations that were calculated on your training set. You want to perform those pre-calculated rotations on your test set. Check out ?predict.prcomp. This will show you how to apply your rotations from your training set onto your test set.

Answer 2

If there are factor levels in your test data that you haven't seen in training data--then, strictly speaking, you cannot predict anything for these observations. You can overcome it by additional assumptions but I am afraid there is no other general solution here than just to give up.

Similar considerations may apply to the other cases. Are you willing to do out-of-sample predictions? What are the underlying assumptions in your model than make it possible? Which role do these assumptions play otherwise, do these influence your results a lot?

That being said, I agree with TBSRounder. You have to use exactly the same transformation as what you applied for the training data.