I have a data-set comprised of a fairly large number of columns (over 1000) relative to the number of rows (370) that I am currently running a random forest regression on. I am a little confused with respect to the best way to go around various tasks such as feature selection and cross-validation.

At present, I am doing the following:

  1. Splitting the data-set 90/10, and running the RF on all features using default hyper-parameters

  2. Using Grid Search to fine tune hyper-parameters and then running an 'optimized' model with these tunings

  3. Using 5-fold cross validation to evaluate the model

  4. Obtaining feature importance scores to indicate which features are performing best

Given the large number of features I have, and the fairly low performance of my models, I would like to find a smarter workflow.

It seems sensible to me to use the feature importance scores to filter out poor performers, but would I do this before or after cross validation? For instance, would I, after completing the above steps and removing features below a certain thrsehold, then run steps 1-3 again? or would I use feature selection before parameter tuning? Do I only cross validate once? Are there any other obvious steps I should integrate?

I appreciate this isn't an exact science, but if there are some industry standards that I'm either getting wrong, or have missed entirely, then I would much appreciate the feedback.


1 Answer 1


370 rows is quite a few, RF does bootstrap but it is still few info. Having too many columns will lead to a more complex model (since the algorithm will work 1 000 dimensions).

Consider doing a pipeline with all the steps and search for hyperparameters and feature selection there. https://scikit-learn.org/stable/modules/generated/sklearn.pipeline.Pipeline.html

You can check a simple tutorial here

In summary, try to build a pipeline with all the steps but maybe the problem is in your data 370 rows is a small sample.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.