I have a data-set comprised of a fairly large number of columns (over 1000) relative to the number of rows (370) that I am currently running a random forest regression on. I am a little confused with respect to the best way to go around various tasks such as feature selection and cross-validation.
At present, I am doing the following:
Splitting the data-set 90/10, and running the RF on all features using default hyper-parameters
Using Grid Search to fine tune hyper-parameters and then running an 'optimized' model with these tunings
Using 5-fold cross validation to evaluate the model
Obtaining feature importance scores to indicate which features are performing best
Given the large number of features I have, and the fairly low performance of my models, I would like to find a smarter workflow.
It seems sensible to me to use the feature importance scores to filter out poor performers, but would I do this before or after cross validation? For instance, would I, after completing the above steps and removing features below a certain thrsehold, then run steps 1-3 again? or would I use feature selection before parameter tuning? Do I only cross validate once? Are there any other obvious steps I should integrate?
I appreciate this isn't an exact science, but if there are some industry standards that I'm either getting wrong, or have missed entirely, then I would much appreciate the feedback.