Nested Cross-Validation with Small dataset

Question

I am currently working with a small dataset (only 175 samples, 45 features) and have been reading on the proper way to cross-validate my model. I had started with a basic cross-validation using a grid search, but have seen that a nested CV structure is superior as it reduces both bias in the model and model selection bias post-CV.

The issue I have is that with a small dataset I am worried about having an insufficiently sized training set in my inner loop since I will need to set aside part of the dataset for inner and out validation. I am using 5-fold CV with 3 repeats as the hyperparameter tuning CV in my inner loop.

Is my concern valid? Any guidance would be appreciated.

Answer 1

First two general observations:

• 45 features for 175 samples looks like many features for few instances. Of course it completely depends on the actual data, the type of features, etc., but at first sight there's a high risk of overifitting.
• There's a confusion about grid search. Cross-validating (CV) is for obtaining a reliable estimate of performance, grid search is for finding the best values for some hyper-parameters. But it's true that it's common to use CV during parameter tuning.

I would suggest to start with a simple CV experiment just to see how much variance there is between folds. If the variance is very high, there is a more serious issue than running nested CV.

5-fold CV means using 80% of the instances as training set every time. My guess would be that it doesn't matter much, in the sense using 140 instances instead of 175 probably has similar effects: either there's no enough data with 175 instances and then also with 140 of course, or there is enough with 175 and in this case it's likely to be ok with 140. But in case there's really a problem with the number of instances (you would notice from the high variance between runs), you can always increase the number of folds.