Splitting and scaling of ML training and test data

Question

I gather you are supposed to split data into training and test before you scale/shift to avoid data leakage. The issue I have with this is how do you cope with values in the test set that are outside of the range of those in the training set? Likewise with the output - you scale the training outputs but the test and predicted outputs could lie outside this range and cause an error with the inverse scaling.

The only solution I can see is to include some wiggle room in the scaler so it can cope with values outside of the training range. Standard scalers like Scikit MinMax don't support this as far as I know.

Answer 1

In an ideal world your train and test data should be from the same distribution. However obviously that is not always possible. E.g. with time series of prices for something, inflation means the distribution is always shifting right, but generally you want test data to come after training data.

There are a few things you can do, depending on what kind of data it is.

Let's get the ugly one out of the way first: you can clip all future inputs to the range of values seen in the training data. I'd only do this if you are actually hitting problems; as others have said, normally it doesn't break anything.

Or, if MinMaxScaler is giving problems, consider using StandardScaler instead.

If you have domain knowledge you might be able to specify the theoretical min/max, rather than learning it from the training data. E.g. maybe a value will always be between 0 and 100, or between -1.0 and +1.0.

If you know the conditions that cause the extreme values, you can actually add them to the training data. For instance in a NLP sentiment model, I could create artificial sentences that demonstrate the extreme emotions.

For data that changes over time, you might be able to use a domain-specific normalization. For instance, for a price database I could change absolute values to be a value relative to what they were a year ago. Train, test, and future values all ought to be in the same distribution now.

Standard scalers like Scikit MinMax don't support [adding wiggle room]

With MinMaxScaler you could set feature_range to 0.01 to 0.99, instead of the default 0.0 to 1.0.

Answer 2

I think there could be slight misunderstanding: you scale both the training data and the testing data. Importantly, you have to make sure that you fit the scaler on the training data; then transform the training data with that fitted scaler, AND transform the testing dataset with that same fitted scaler.

As Dave's comment says, it wouldn't matter if there are values in the testing set that are outside of the "range" of the training set; that wouldn't cause an error.