0
$\begingroup$

I have come here from this great answer. I have come across many approaches for using cross validation and the answer to the attached question is by far explaining it the best to me. My dilemma is that now that I m not able to figure out what to use Kfold cross validation for:-

  1. Testing overfitting?
  2. Hyperparam tuning?
  3. Any other use case?

and that too how? I am unable to figure out what to do with the average score that comes after kfold cross val, what to do with the folds and what to do with a model trained on k-1 folds of train data?

$\endgroup$
1
$\begingroup$

Answering the "what to do" point, if you use the scikit-learn GridSearchCV class (from sklearn.model_selection), you can get from it the following:

  • best params found among the ones you enter with the 'param_grid' input, based on the 'scoring' metric you want (i.e. roc_auc, recall...)
  • and the most important point, you can directly access the best estimator (i.e. the model instanced with the best hyperparms found in the CV process) already refit with the whole training dataset.

I have seen some source which make a "manual" retrain on the whole training set, but it is not necessary as scikit-learn already let's you accessing the refit model on the whole train set :)

$\endgroup$
  • $\begingroup$ Okay so here it goes! now that you have mentioned, I am not able to understand how grid search cv got into picture here. I mean, what happened to kfold? Are we not using it anymore?. I understand what grid search and k fold do separately but I cant understand what the flow around them is. $\endgroup$ – Dev_Man Oct 22 at 5:37
  • $\begingroup$ Thanks for your answer; I pointed out grid search CV for the main use case when using k-folds strategy for tuning among a list of hyperparameters; I mean, once you have run the Grid Search CV, you can directly access the best found model (based in the best average selected metric score) refit on the whole dataset, so you can forget about the 'mini-models' trained on k-1 folds $\endgroup$ – German C M Oct 22 at 8:20
  • $\begingroup$ so does that mean that kfold and grid search cv are clubbed together? If yes is there any example you could should me via a link which demonstrates this? Thanks! $\endgroup$ – Dev_Man Oct 22 at 9:54
  • 1
    $\begingroup$ here you go with the code, both in .py and .ipynb versions, feel free to use it: github.com/GermanCM/machine_learning_concepts_checks ; a vote up is welcome ;) $\endgroup$ – German C M Oct 22 at 17:39
1
$\begingroup$

Cross validation is basically for hyper-parameter tuning.

You train a set of model hyper-parameter setting on kfold cross validation and take the average score from kfold cross validation as the approximate performance of each model hyper-parameter setting. Then, the model hyper-parameter setting which has the highest score will be the choice of your model setting. This hyper-parameter setting can be treat as the best you can get from this kind of model.

Later, you can use this model setting to test the general performance on your test dataset.

$\endgroup$
  • $\begingroup$ Okay. But then why don't we do another train test split from the train data for hyper parameter tuning? What good does Kfold bring to the table? $\endgroup$ – Dev_Man Oct 22 at 5:11
  • $\begingroup$ It actually depends on the size of your dataset. For training, we would like to have as more data points as possible for training phase in order to better model the real behavior of data. If your dataset is large enough, say 1M data points, then train test split will be better than cross validation. In the other hand, if you have smaller dataset, say server k data points, the training set will be not big enough. So, cross validation is the solution to the problems with smaller datasets. $\endgroup$ – 1tan Oct 22 at 5:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.