I'm currently working with Python and Scikit learn for classification purposes, and doing some reading around GridSearch I thought this was a great way for optimising my estimator parameters to get the best results.
My methodology is this:
- Split my data into training/test.
- Use GridSearch with 5Fold Cross validation to train and test my estimators(Random Forest, Gradient Boost, SVC amongst others) to get the best estimators with the optimal combination of hyper parameters.
- I then calculate metrics on each of my estimators such as Precision, Recall, FMeasure and Matthews Correlation Coefficient, using my test set to predict the classifications and compare them to actual class labels.
It is at this stage that I see strange behaviour and I'm unsure how to proceed. Do I take the .best_estimator_ from the GridSearch and use this as the 'optimal' output from the grid search, and perform prediction using this estimator? If I do this I find that the stage 3 metrics are usually much lower than if I simply train on all training data and test on the test set. Or, do I simply take the output GridSearchCV object as the new estimator? If I do this I get better scores for my stage 3 metrics, but it seems odd using a GridSearchCV object instead of the intended classifier (E.g. a random Forest) ...
EDIT: So my question is what is the difference between the returned GridSearchCV object and the .best_estimator_ attribute? Which one of these should I use for calculating further metrics? Can I use this output like a regular classifier (e.g. using predict), or else how should I use it?