0
$\begingroup$

While trying to evaluate my Ridge Regression model and using GridSearchCV to find the best parameter. I noticed that the best estimator changes every time I change the random_state in my KFold object (cv parameter). With this in mind how do I choose the most optimal hyper parameter to implement my model.

$\endgroup$
2
  • 2
    $\begingroup$ Which hyperparameter(s) are you tuning? Are the values radically different depending on random_state, or are they mostly similar with small variations? $\endgroup$
    – zachdj
    Apr 20, 2020 at 16:30
  • $\begingroup$ @zachdj I am trying to tune the alpha parameter in Ridge Regression. As I run GridSearchCV with different random_states for Kfold I get really varying best_estimator_ value for alpha ranging from 3 to 25. I am getting a r2 score between 0.6 to 0.7. I think I should mention that my data is pretty small (300 entries). Is that $\endgroup$
    – Ahmed Jyad
    Apr 21, 2020 at 16:27

2 Answers 2

0
$\begingroup$

I am assuming, you are taking about the random_state of the Model, not the GridSearch (as it doesn't have a random_state) RandomizedSearchCV do need one.

A random state defines a starting point for an underlying random process.

With an ideal model, the difference should be very small. Though it will be. If it is high, it means there is some Outlier data point/pattern due to which certain value is favoured.

Ignore - if small

If it is large for a specific value
- Check the data
- Increase K for the CrossVal

From another Stackexchange Question

random state values which performed well in the validation set do not correspond to those which would perform well in a new, unseen test set.

From MachineLearning Mastery

We can increase k and build even more models, as long as the data within each fold remains representative of the problem

Links
Stochastic Process
Randomness in ML
Is-random-state-a-parameter-to-tune

$\endgroup$
2
  • $\begingroup$ The random_state is in the cv attribute of GridSearchCV. I used Kfold to shuffle my data. $\endgroup$
    – Ahmed Jyad
    Apr 21, 2020 at 16:48
  • $\begingroup$ @AhmedJyad, that's very relevant information; please add it to the question. (And then we can delete these two comments.) $\endgroup$
    – Ben Reiniger
    Jan 22, 2021 at 16:51
0
$\begingroup$

If the scoring is very dependent on random_state, it would be better to try to address that rather than choosing a hyperparameter from what you have. You mentioned you used KFold, and that your data is quite small; I suggest trying RepeatedKFold instead.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.