1
$\begingroup$

I am currently trying to optimise some parameters on my model (15000 samples). What I am finding is a relatively large variance in the loss function 2%-10% which makes it hard to identify which parameter is the best. This appears to happen based on how the random number generator splits the data into train/test sets.

I have tried :

  • CV 5-fold

  • Split of 75%

Fixing the random seed does help (or using the same test set), but it concerns me that I get such variations based on what samples are in the test set. It seems alarming that the 'best parameter' is so dependent on a particular shuffle of the data and I worry how it translates to real world use.

What are people's approach to situations like this? I was thinking I could just repeat each test multiple times and take the average, but that has very large computation costs and seems very inefficient.

$\endgroup$
1
$\begingroup$

As mentioned it can be a good idea to repeat CV a few times and average the results to obtain a more reliable estimate

If you find many parameter constellations that are within one standard deviation (or in that neighborhood) of the best performing model, it can make sense to choose a model with a slightly worse CV performance, but with a simpler decision boundary (e.g. shallower decision trees, smaller gammas in an RBF-kernel SVM, stronger regularization parameters etc.) - something that was suggested for example here.

$\endgroup$
  • $\begingroup$ That is a very interesting approach to the problem. Has opened my mind! $\endgroup$ – simeon Aug 14 '17 at 4:52
1
$\begingroup$

I'm afraid you have to repeat the K-fold CV a few times (with different seeds each time) and average the results. I'd guess that the high variance comes from the small size of the dataset.

$\endgroup$
1
$\begingroup$
  • Averaging might help. You can optimize your hyper parameter tuning time using Bayesian Hyper-parameter tuning approach.

  • Try to reduce the number of features or try algorithms like Random Forest with other standard techniques like k-fold repeated CV.

  • Try creating new features which might help you differentiate classes

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.