25
$\begingroup$

I'm currently working with Python and Scikit learn for classification purposes, and doing some reading around GridSearch I thought this was a great way for optimising my estimator parameters to get the best results.

My methodology is this:

  1. Split my data into training/test.
  2. Use GridSearch with 5Fold Cross validation to train and test my estimators(Random Forest, Gradient Boost, SVC amongst others) to get the best estimators with the optimal combination of hyper parameters.
  3. I then calculate metrics on each of my estimators such as Precision, Recall, FMeasure and Matthews Correlation Coefficient, using my test set to predict the classifications and compare them to actual class labels.

It is at this stage that I see strange behaviour and I'm unsure how to proceed. Do I take the .best_estimator_ from the GridSearch and use this as the 'optimal' output from the grid search, and perform prediction using this estimator? If I do this I find that the stage 3 metrics are usually much lower than if I simply train on all training data and test on the test set. Or, do I simply take the output GridSearchCV object as the new estimator? If I do this I get better scores for my stage 3 metrics, but it seems odd using a GridSearchCV object instead of the intended classifier (E.g. a random Forest) ...

EDIT: So my question is what is the difference between the returned GridSearchCV object and the .best_estimator_ attribute? Which one of these should I use for calculating further metrics? Can I use this output like a regular classifier (e.g. using predict), or else how should I use it?

$\endgroup$
29
$\begingroup$

Decided to go away and find the answers that would satisfy my question, and write them up here for anyone else wondering.

The .best_estimator_ attribute is an instance of the specified model type, which has the 'best' combination of given parameters from the param_grid. Whether or not this instance is useful depends on whether the refit parameter is set to True (it is by default). For example:

clf = GridSearchCV(estimator=RandomForestClassifier(), 
                    param_grid=parameter_candidates,
                    cv=5,
                    refit=True,
                    error_score=0,
                    n_jobs=-1)

clf.fit(training_set, training_classifications)
optimised_random_forest = clf.best_estimator_
return optimised_random_forest

Will return a RandomForestClassifier. This is all pretty clear from the documentation. What isn't clear from the documentation is why most examples don't specifically use the .best_estimator_ and instead do this:

clf = GridSearchCV(estimator=RandomForestClassifier(), 
                    param_grid=parameter_candidates,
                    cv=5,
                    refit=True,
                    error_score=0,
                    n_jobs=-1)

clf.fit(training_set, training_classifications)
return clf

This second approach returns a GridSearchCV instance, with all the bells and whistles of the GridSearchCV such as .best_estimator_, .best_params, etc, which itself can be used like a trained classifier because:

Optimised Random Forest Accuracy:  0.916970802919708
[[139  47]
 [ 44 866]]
GridSearchCV Accuracy:  0.916970802919708
[[139  47]
 [ 44 866]]

It just uses the same best estimator instance when making predictions. So in practise there's no difference between these two unless you specifically only want the estimator instance itself. As a side note, my differences in metrics were unrelated and down to a buggy class weighting function.

$\endgroup$
  • $\begingroup$ Thanks for your post @Dan, it is very helpful. I wanted to ask you for one clarification. At the latter case, if I have refit=False then clf.fit won't be done with the best classifier? $\endgroup$ – Poete Maudit Sep 28 '18 at 9:42
  • $\begingroup$ @PoeteMaudit The refit parameter tells the GridSearchCV function to take the best parameters found, and retrain the model using those parameters on the whole dataset. If refit=False, then best_estimator is not available, according to the documentation: scikit-learn.org/stable/modules/generated/… $\endgroup$ – Dan Carter Mar 13 '19 at 9:51
0
$\begingroup$

GridSearchCV lets you combine an estimator with a grid search preamble to tune hyper-parameters. The method picks the optimal parameter from the grid search and uses it with the estimator selected by the user. GridSearchCV inherits the methods from the classifier, so yes, you can use the .score, .predict, etc.. methods directly through the GridSearchCV interface. If you wish to extract the best hyper-parameters identified by the grid search you can use .best_params_ and this will return the best hyper-parameter. You can then pass this hyper-parameter to your estimator separately.

Using .predict directly will yield the same results as getting the best hyper-parameter through .best_param_ and then using it in your model. By understanding the underlining workings of grid search we can see why this is the case.


Grid Search

This technique is used to find the optimal parameters to use with an algorithm. This is NOT the weights or the model, those are learned using the data. This is obviously quite confusing so I will distinguish between these parameters, by calling one hyper-parameters.

Hyper-parameters are like the k in k-Nearest Neighbors (k-NN). k-NN requires the user to select which neighbor to consider when calculating the distance. The algorithm then tunes a parameter, a threshold, to see if a novel example falls within the learned distribution, this is done with the data.

How do we choose k?

Some people simply go with recommendations based on past studies of the data type. Others use grid search. This method will be able to best determine which k is the optimal to use for your data.

How does it work?

First you need to build a grid. This is essentially a set of possible values your hyper-parameter can take. For our case we can use $[1, 2, 3, ..., 10]$. Then you will train your k-NN model for each value in the grid. First you would do 1-NN, then 2-NN, and so on. For each iteration you will get a performance score which will tell you how well your algorithm performed using that value for the hyper-parameter. After you have gone through the entire grid you will select the value that gave the best performance.

This goes against the principles of not using test data!!

You would be absolutely right. That is the reason grid search is often mixed with cross-validation. So that we keep the test data completely separate until we are truly satisfied with our results and are ready to test. $n$-fold cross-validation takes a training set and separates it into $n$ parts. It then trains on $n-1$ folds and tests on the fold which was left out. For each value in the grid, the algorithm will be retrained $n$ times, for each fold being left out. Then the performance across each fold is averaged and that is the achieved performance for that hyper-parameter value.

The selected hyper-parameter value is the one which achieves the highest average performance across the n-folds. Once you are satisfied with your algorithm, then you can test it on the testing set. If you go straight to the testing set then you are risking overfitting.

$\endgroup$
  • $\begingroup$ Hi Jah, this is a good answer but I'm still none the wiser as to the answer of my question. I've updated the question title and the question itself to try and make things more clear. $\endgroup$ – Dan Carter Aug 1 '17 at 13:59
  • $\begingroup$ Write your own grid search. It's literally make an array, then add a for loop around your model. Then at the end of your for loop record the resulting performance into an array. After you have gone through all the possible values in your grid, look at the arrays of performances, and pick out the best one. That is the optimal value for your hyper-parameter. Relying on built-in functions for the basics is highly not recommended for data science. Data varies so wildly and its best for you to have control! $\endgroup$ – JahKnows Aug 1 '17 at 14:03
  • $\begingroup$ That would be a good suggestion if I only had one hyperparameter to optimise, but if I have 4? 5? A 4/5 times nested for loop is ugly and I see no need to reinvent the wheel here, that would be a waste of time, and it's the reason packages like this exist. $\endgroup$ – Dan Carter Aug 1 '17 at 14:05
  • $\begingroup$ GridSearchCV lets you combine an estimator with GridSearchCV setting. So it does exactly what we just discussed. It then picks the optimal parameter and uses it with the estimator you selected. GridSearchCV inherits the methods from the classifier, so yes, you can use the .score, .predict, etc.. methods directly through the GridSearchCV interface. I do not recommend doing this however, easier tools means less control. For something so simple as a grid search just code it yourself. $\endgroup$ – JahKnows Aug 1 '17 at 14:09
  • 1
    $\begingroup$ This answer does not address the question, which is regarding GridSearchCV usage. $\endgroup$ – Hobbes Aug 1 '17 at 17:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.