0
$\begingroup$

manually optimizing parameter

using GridSearchCV to optimize parameter

I am trying out hyperparameter tuning vs manually selecting the best parameter (max_leaf_nodes) on a decision tree model with mean absolute error as the scoring. In theory, both should give me the same MAE and max_leaf_nodes; but, both are giving me different MAEs. Also, if I change the value of cv in GridSearchCV I get different results. So basically I have two questions:

  1. Why am I getting different max_leaf_nodes and MAE in both cases?

  2. How do I determine the value of cv in GridsearchCV, because I get different results for cv = 3, cv = 5, and cv = 10?

$\endgroup$
6
  • $\begingroup$ I am not sure how big the difference is, but since you are not specifying a seed in the second example the results will differ depending on the seed. $\endgroup$
    – Oxbowerce
    Jul 1 at 17:28
  • $\begingroup$ I tried setting the seed to 69 everywhere. But still I'm getting different values for mean absolute error and max_leaf_nodes. When performing GridSearchCV mae= 27378 and best max_leaf_nodes = 100 but whne manually optimizing mae = 28202 and max_leaf_nodes = 50! $\endgroup$
    – spectre
    Jul 1 at 18:03
  • $\begingroup$ That might be explained by the default scorer that is used when using GridSearchCV, see also this stackexchange answer. $\endgroup$
    – Oxbowerce
    Jul 1 at 18:17
  • $\begingroup$ Please do not insert images of code; paste the code into a code-formatted block instead. $\endgroup$ Jul 1 at 18:31
  • $\begingroup$ @Oxbowerce I've used the same scoring (mean absolute error) in both the cases. Also the seeds are same everywhere. Still not getting similar results! $\endgroup$
    – spectre
    Jul 2 at 5:42
0
$\begingroup$

Your manual approach gives the MAE on the test set. Because you've set an integer for the parameter cv, the GridSearchCV is doing k-fold cross-validation (see the parameter description in grid search docs), and so the score .best_score_ is the average MAE on the multiple test folds.

If you really want a single train/test split, you can do that in GridSearchCV, see e.g. this SO post.

$\endgroup$
1
  • $\begingroup$ So if I understand correctly, in the first case, I am getting mae based on only the "test set" but when using gridsearch I am getting mae based on the multiple sets of "train set" since I set cv = 5? Am I correct? $\endgroup$
    – spectre
    Jul 2 at 6:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.