Random Forest workflow?

Question

I have a data-set comprised of a fairly large number of columns (over 1000) relative to the number of rows (370) that I am currently running a random forest regression on. I am a little confused with respect to the best way to go around various tasks such as feature selection and cross-validation.

At present, I am doing the following:

1. Splitting the data-set 90/10, and running the RF on all features using default hyper-parameters

2. Using Grid Search to fine tune hyper-parameters and then running an 'optimized' model with these tunings

3. Using 5-fold cross validation to evaluate the model

4. Obtaining feature importance scores to indicate which features are performing best

Given the large number of features I have, and the fairly low performance of my models, I would like to find a smarter workflow.

It seems sensible to me to use the feature importance scores to filter out poor performers, but would I do this before or after cross validation? For instance, would I, after completing the above steps and removing features below a certain thrsehold, then run steps 1-3 again? or would I use feature selection before parameter tuning? Do I only cross validate once? Are there any other obvious steps I should integrate?

I appreciate this isn't an exact science, but if there are some industry standards that I'm either getting wrong, or have missed entirely, then I would much appreciate the feedback.

Answer 1

370 rows is quite a few, RF does bootstrap but it is still few info. Having too many columns will lead to a more complex model (since the algorithm will work 1 000 dimensions).

Consider doing a pipeline with all the steps and search for hyperparameters and feature selection there. https://scikit-learn.org/stable/modules/generated/sklearn.pipeline.Pipeline.html

You can check a simple tutorial here

In summary, try to build a pipeline with all the steps but maybe the problem is in your data 370 rows is a small sample.