Can one perform Feature Selection on a subset of training data?

Question

I have a training data set with almost one million rows and I am considering eight features initially. My machine learning model will be Random Forest regressor.
In Section 3.4.7 of "Feature Engineering and Selection: A Practical Approach for Predictive Models" there is this:

Finally, with large amounts of data, alternative data usage schemes might be a good idea. Instead of a simple training/test split, multiple splits can be created for specific purposes. For example, a specific split of the data can be used to determine the relevant predictors for the model prior to model building using the training set. This would reduce the need to include the expensive feature selection steps inside of resampling.

Based on this and for finding relevant features, I want to do an exhaustive search over all possible subsets of initial set of features on 10% of my training data (~100k rows). The evaluation of each subset will be done by 5fold cross-validating random forest with the same hyperparameters for all of the subsets.

Why 10 %? ans: to reduce computation time and because I think 100k rows would give me a reliable set of features. I have no reference to support my choices. Please let me know if you have one.

After finding THE best subset, I go on and do hyperparameter tuning using 100% of my training data.

Am I doing something wrong? Could you suggest me some references to look for vis-a-vis the parts that I am doing wrong?

I searched about this "specific split" that Max Kuhn talks about, couldn't find any similar suggestion.

More details:

Why do I want to do feature selection if I only have eight features, especially considering that RF is somewhat insensitive to redundant/irrelevant features (with respect to predictive performance, not feature importance/selection)?

Apart from the general idea of decreasing the computational time by using less features, the features are meteorological variables that may not be available/accessible easily and everywhere. So if I am to build a model that can be used in other case studies and research, it would make it easier to use if it has less variables.

Using RF for feature selection inside another method is kind of more useful I think because it can also consider the interaction effects implicitly.

I want to do all these things in an academic work, that's why I need to justify all the steps and decisions that I make. Here I am mostly concerned about the fact that I am using only 100k of my rows for feature selection. One might say, why 100k? why not 200k? and etc.

Another doubt of mine is: since I do a supervised feature selection, I should do the tuning on the remaining 90% of training data, not on 100%. Do you agree?

Answer 1

After finding THE best subset doesn't make sense. Your approach is not correct in this way.

If you cannot handle all data you may have to follow one of approaches below:

1- Perform clustering and chose 10% (the more the better, based on what your computer can handle) from each cluster so all 10 subsets may have similar distribution. As a result, the models theoretically should have similar performance. This approach is useful if you can find clusters in your data. Don't forget to split each subset to training/test subsets!

2- If no clusters found, develop 10 models for each subset and use the average of outputs (bagging), and see how it works.

Hyper parameter tuning is an expensive task, even for small datasets; so it may not be a good idea to do it using not 100% but 70-80% of your data as the training set.

Another thing that you should consider is the ML algorithm you choose. Random Forest may or may not be a good choice. So try different algorithms on small subsets to have an idea which algorithm may work better for your problem. Then think about hyper parameter tuning and feature selection.