How to use the output of GridSearch?

Question

I'm currently working with Python and Scikit learn for classification purposes, and doing some reading around GridSearch I thought this was a great way for optimising my estimator parameters to get the best results.

My methodology is this:

1. Split my data into training/test.
2. Use GridSearch with 5Fold Cross validation to train and test my estimators(Random Forest, Gradient Boost, SVC amongst others) to get the best estimators with the optimal combination of hyper parameters.
3. I then calculate metrics on each of my estimators such as Precision, Recall, FMeasure and Matthews Correlation Coefficient, using my test set to predict the classifications and compare them to actual class labels.

It is at this stage that I see strange behaviour and I'm unsure how to proceed. Do I take the .best_estimator_ from the GridSearch and use this as the 'optimal' output from the grid search, and perform prediction using this estimator? If I do this I find that the stage 3 metrics are usually much lower than if I simply train on all training data and test on the test set. Or, do I simply take the output GridSearchCV object as the new estimator? If I do this I get better scores for my stage 3 metrics, but it seems odd using a GridSearchCV object instead of the intended classifier (E.g. a random Forest) ...

EDIT: So my question is what is the difference between the returned GridSearchCV object and the .best_estimator_ attribute? Which one of these should I use for calculating further metrics? Can I use this output like a regular classifier (e.g. using predict), or else how should I use it?

Answer 1

Decided to go away and find the answers that would satisfy my question, and write them up here for anyone else wondering.

The .best_estimator_ attribute is an instance of the specified model type, which has the 'best' combination of given parameters from the param_grid. Whether or not this instance is useful depends on whether the refit parameter is set to True (it is by default). For example:

clf = GridSearchCV(estimator=RandomForestClassifier(), 
                    param_grid=parameter_candidates,
                    cv=5,
                    refit=True,
                    error_score=0,
                    n_jobs=-1)

clf.fit(training_set, training_classifications)
optimised_random_forest = clf.best_estimator_
return optimised_random_forest

Will return a RandomForestClassifier. This is all pretty clear from the [documentation][1]. What isn't clear from the documentation is why most examples don't specifically use the .best_estimator_ and instead do this:

clf = GridSearchCV(estimator=RandomForestClassifier(), 
                    param_grid=parameter_candidates,
                    cv=5,
                    refit=True,
                    error_score=0,
                    n_jobs=-1)

clf.fit(training_set, training_classifications)
return clf

This second approach returns a GridSearchCV instance, with all the bells and whistles of the GridSearchCV such as .best_estimator_, .best_params, etc, which itself can be used like a trained classifier because:

Optimised Random Forest Accuracy:  0.916970802919708
[[139  47]
 [ 44 866]]
GridSearchCV Accuracy:  0.916970802919708
[[139  47]
 [ 44 866]]

It just uses the same best estimator instance when making predictions. So in practise there's no difference between these two unless you specifically only want the estimator instance itself. As a side note, my differences in metrics were unrelated and down to a buggy class weighting function. [1]: http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html#sklearn.model_selection.GridSearchCV.fit

Answer 2

GridSearchCV lets you combine an estimator with a grid search preamble to tune hyper-parameters. The method picks the optimal parameter from the grid search and uses it with the estimator selected by the user. GridSearchCV inherits the methods from the classifier, so yes, you can use the .score, .predict, etc.. methods directly through the GridSearchCV interface. If you wish to extract the best hyper-parameters identified by the grid search you can use .best_params_ and this will return the best hyper-parameter. You can then pass this hyper-parameter to your estimator separately.

Using .predict directly will yield the same results as getting the best hyper-parameter through .best_param_ and then using it in your model. By understanding the underlining workings of grid search we can see why this is the case.

---

Grid Search

This technique is used to find the optimal parameters to use with an algorithm. This is NOT the weights or the model, those are learned using the data. This is obviously quite confusing so I will distinguish between these parameters, by calling one hyper-parameters.

Hyper-parameters are like the k in k-Nearest Neighbors (k-NN). k-NN requires the user to select which neighbor to consider when calculating the distance. The algorithm then tunes a parameter, a threshold, to see if a novel example falls within the learned distribution, this is done with the data.

How do we choose k?

Some people simply go with recommendations based on past studies of the data type. Others use grid search. This method will be able to best determine which k is the optimal to use for your data.

How does it work?

First you need to build a grid. This is essentially a set of possible values your hyper-parameter can take. For our case we can use $[1, 2, 3, ..., 10]$. Then you will train your k-NN model for each value in the grid. First you would do 1-NN, then 2-NN, and so on. For each iteration you will get a performance score which will tell you how well your algorithm performed using that value for the hyper-parameter. After you have gone through the entire grid you will select the value that gave the best performance.

This goes against the principles of not using test data!!

You would be absolutely right. That is the reason grid search is often mixed with cross-validation. So that we keep the test data completely separate until we are truly satisfied with our results and are ready to test. $n$-fold cross-validation takes a training set and separates it into $n$ parts. It then trains on $n-1$ folds and tests on the fold which was left out. For each value in the grid, the algorithm will be retrained $n$ times, for each fold being left out. Then the performance across each fold is averaged and that is the achieved performance for that hyper-parameter value.

The selected hyper-parameter value is the one which achieves the highest average performance across the n-folds. Once you are satisfied with your algorithm, then you can test it on the testing set. If you go straight to the testing set then you are risking overfitting.