Does it help to have similar values for features in train and test data to make accurate predictions?

Question

I am quite new to some concepts of machine learning and having hard time understanding the following.

Suppose I have a supervised classifier (random forest) trained with a dataset with several features.

Do the features in the test dataset need to have values that are somewhat similar (or closer) to the training data (or in the same domain).

For example, take training data record: <'label A', 12, 23, 3412, 65> (assume other 'label A' types are similar to this, with only +-10 difference for each feature), test data: X: <10, 21, 3000, 80> and Y: <0.12. 0.23, 34.12, 0.65>.

Out of X and Y, which has a higher chance of being classified as type 'label A'?

Please make a note of any assumptions you make.

Answer 1

This is both a simple and complex concept and one that we are always concerned with while building models.

The short answer is, the data in your training and test sets needs to be randomly selected, and accordingly you have no control over the range and variation in either set relative to a fixed amount of modeling data.

The long answer is that any model will interpolate better than it will extrapolate because it is easier to describe what is known by the model. So, if there are a lot of values outside of the training range, it will for sure affect the predictive capabilities of your resulting model.

The nuances of this depend on a lot of things, how much data you have, how divergent the training and test variables are, what specific model tuning parameters you use how much the outcome variable varies with its predictors, why and how.

But ultimately the training set should be large enough and varied enough such that it captures all of the variance in the outcome variable you hope to predict. This more than likely means that your features should be fully described at all possible values they might take.