I have a training data set with almost one million rows and I am considering eight features initially. My machine learning model will be Random Forest regressor.
In Section 3.4.7 of "Feature Engineering and Selection: A Practical Approach for Predictive Models" there is this:
Finally, with large amounts of data, alternative data usage schemes might be a good idea. Instead of a simple training/test split, multiple splits can be created for specific purposes. For example, a specific split of the data can be used to determine the relevant predictors for the model prior to model building using the training set. This would reduce the need to include the expensive feature selection steps inside of resampling.
Based on this and for finding relevant features, I want to do an exhaustive search over all possible subsets of initial set of features on 10% of my training data (~100k rows). The evaluation of each subset will be done by 5fold cross-validating random forest with the same hyperparameters for all of the subsets.
Why 10 %? ans: to reduce computation time and because I think 100k rows would give me a reliable set of features. I have no reference to support my choices. Please let me know if you have one.
After finding THE best subset, I go on and do hyperparameter tuning using 100% of my training data.
Am I doing something wrong? Could you suggest me some references to look for vis-a-vis the parts that I am doing wrong?
I searched about this "specific split" that Max Kuhn talks about, couldn't find any similar suggestion.
Why do I want to do feature selection if I only have eight features, especially considering that RF is somewhat insensitive to redundant/irrelevant features (with respect to predictive performance, not feature importance/selection)?
Apart from the general idea of decreasing the computational time by using less features, the features are meteorological variables that may not be available/accessible easily and everywhere. So if I am to build a model that can be used in other case studies and research, it would make it easier to use if it has less variables.
Using RF for feature selection inside another method is kind of more useful I think because it can also consider the interaction effects implicitly.
I want to do all these things in an academic work, that's why I need to justify all the steps and decisions that I make. Here I am mostly concerned about the fact that I am using only 100k of my rows for feature selection. One might say, why 100k? why not 200k? and etc.
Another doubt of mine is: since I do a supervised feature selection, I should do the tuning on the remaining 90% of training data, not on 100%. Do you agree?