Grid Search and High Variance

Question

I am currently trying to optimise some parameters on my model (15000 samples). What I am finding is a relatively large variance in the loss function 2%-10% which makes it hard to identify which parameter is the best. This appears to happen based on how the random number generator splits the data into train/test sets.

I have tried :

• CV 5-fold

• Split of 75%

Fixing the random seed does help (or using the same test set), but it concerns me that I get such variations based on what samples are in the test set. It seems alarming that the 'best parameter' is so dependent on a particular shuffle of the data and I worry how it translates to real world use.

What are people's approach to situations like this? I was thinking I could just repeat each test multiple times and take the average, but that has very large computation costs and seems very inefficient.

Answer 1

As mentioned it can be a good idea to repeat CV a few times and average the results to obtain a more reliable estimate

If you find many parameter constellations that are within one standard deviation (or in that neighborhood) of the best performing model, it can make sense to choose a model with a slightly worse CV performance, but with a simpler decision boundary (e.g. shallower decision trees, smaller gammas in an RBF-kernel SVM, stronger regularization parameters etc.) - something that was suggested for example here.

Answer 2

I'm afraid you have to repeat the K-fold CV a few times (with different seeds each time) and average the results. I'd guess that the high variance comes from the small size of the dataset.

Answer 3

• Averaging might help. You can optimize your hyper parameter tuning time using Bayesian Hyper-parameter tuning approach.

• Try to reduce the number of features or try algorithms like Random Forest with other standard techniques like k-fold repeated CV.

• Try creating new features which might help you differentiate classes