0
$\begingroup$

manually optimizing parameter

using GridSearchCV to optimize parameter

I am trying out hyperparameter tuning vs manually selecting the best parameter (max_leaf_nodes) on a decision tree model with mean absolute error as the scoring. In theory, both should give me the same MAE and max_leaf_nodes; but, both are giving me different MAEs. Also, if I change the value of cv in GridSearchCV I get different results. So basically I have two questions:

  1. Why am I getting different max_leaf_nodes and MAE in both cases?

  2. How do I determine the value of cv in GridsearchCV, because I get different results for cv = 3, cv = 5, and cv = 10?

Improve this question
$\endgroup$
6
  • $\begingroup$ I am not sure how big the difference is, but since you are not specifying a seed in the second example the results will differ depending on the seed. $\endgroup$
    – Oxbowerce
    Commented Jul 1, 2021 at 17:28
  • $\begingroup$ I tried setting the seed to 69 everywhere. But still I'm getting different values for mean absolute error and max_leaf_nodes. When performing GridSearchCV mae= 27378 and best max_leaf_nodes = 100 but whne manually optimizing mae = 28202 and max_leaf_nodes = 50! $\endgroup$
    – spectre
    Commented Jul 1, 2021 at 18:03
  • $\begingroup$ That might be explained by the default scorer that is used when using GridSearchCV, see also this stackexchange answer. $\endgroup$
    – Oxbowerce
    Commented Jul 1, 2021 at 18:17
  • $\begingroup$ Please do not insert images of code; paste the code into a code-formatted block instead. $\endgroup$
    – Ben Reiniger ♦
    Commented Jul 1, 2021 at 18:31
  • $\begingroup$ @Oxbowerce I've used the same scoring (mean absolute error) in both the cases. Also the seeds are same everywhere. Still not getting similar results! $\endgroup$
    – spectre
    Commented Jul 2, 2021 at 5:42

1 Answer 1

Reset to default
0
$\begingroup$

Your manual approach gives the MAE on the test set. Because you've set an integer for the parameter cv, the GridSearchCV is doing k-fold cross-validation (see the parameter description in grid search docs), and so the score .best_score_ is the average MAE on the multiple test folds.

If you really want a single train/test split, you can do that in GridSearchCV, see e.g. this SO post.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ So if I understand correctly, in the first case, I am getting mae based on only the "test set" but when using gridsearch I am getting mae based on the multiple sets of "train set" since I set cv = 5? Am I correct? $\endgroup$
    – spectre
    Commented Jul 2, 2021 at 6:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.