GridSearch without CV

Question

I create a Random Forest and Gradient Boosting Regressor by using GridSearchCV. For the Gradient Boosting Regressor, it takes too long for me. But I need to know which are the best parameters for the models. So I am thinking if there is a GridSearch without CV because the OOB score is sufficient to evaluate the models. I hope I could explain what I mean.

Answer 1

GridSearchCV is built around cross validation, but if speed is your main concern, you may be able to get better performance using a smaller number of folds.

From the docs:

class sklearn.model_selection.GridSearchCV(estimator, param_grid, scoring=None, n_jobs=None, iid='deprecated', refit=True, cv=None, verbose=0, pre_dispatch='2*n_jobs', error_score=nan, return_train_score=False)

cv: int, cross-validation generator or an iterable, optional

Determines the cross-validation splitting strategy. Possible inputs for cv are:

    None, to use the default 5-fold cross validation,

    integer, to specify the number of folds in a (Stratified)KFold,

    CV splitter,

    An iterable yielding (train, test) splits as arrays of indices.

For integer/None inputs, if the estimator is a classifier and y is either binary or multiclass, StratifiedKFold is used. In all other cases, KFold is used.

cv defaults to 5, so changing it to 2 should provide a significant speedup for you. This will weaken the cross validation significantly.

Alternatively, you may be able to pass in a single test/train split for your value of cv. This would effectively disable cross validation and remove the benefits that it provides.

Answer 2

By passing a callable for parameter scoring, that uses the model's oob score directly and completely ignores the passed data, you should be able to make the GridSearchCV act the way you want it to. Just pass a single split for the cv parameter, as @jncranton suggests; you can even go further and make that single split use all the data for the training portion, and the testing portion won't even get used in the above setup. (Does sklearn perform a check to prevent passing cv=1?)

I haven't had a chance to try this out yet:

def oob_scorer(estimator, X, y):
    return estimator.oob_score_

model = GridSearchCV(estimator=RandomForest(...),
                     param_grid={...},
                     scoring=oob_scorer,
                     cv=PredefinedSplit([-1]*TRAIN_SET.shape[0]),
                     ...
                     )

scikit docs:
Fixed split
Custom scorer

Related Qs:
Scikitlearn grid search random forest using oob as metric?
RandomForestClassifier OOB scoring method

I'm not sure the hackiness of this approach is worth it; it wouldn't be terribly difficult to make the grid loop yourself, even with parallelization.

---

EDIT: Yes, a cv-splitter with no test group fails. Hackier by the minute, but you can split off just a single test point, or add a dummy test set, or...
Here's a working example. It does seem the oob_score is being used, and the test set has just a single sacrificial point: https://github.com/bmreiniger/datascience.stackexchange/blob/master/GridSearchNoCV_oob.ipynb

Answer 3

Alternatively, just implement a simple Grid Search algorithm yourself. The book "Introduction to Machine Learning with Python" by Mueller and Guido includes an example using an SVC:

# naive grid search implementation
from sklearn.svm import SVC

X_train, X_test, y_train, y_test = train_test_split(iris.data, iris.target, random_state=0)
print("Size of training set: {} size of test set: {}".format( X_train.shape[0], X_test.shape[0]))

best_score = 0

for gamma in [0.001, 0.01, 0.1, 1, 10, 100]:
    for C in [0.001, 0.01, 0.1, 1, 10, 100]:
        # for each combination of parameters, train an SVC
        svm = SVC(gamma=gamma, C=C)
        svm.fit(X_train, y_train)
        # evaluate the SVC on the test set
        score = svm.score(X_test, y_test)
        # if we got a better score, store the score and parameters
        if score > best_score:
            best_score = score
            best_parameters = {'C': C, 'gamma': gamma}

print("Best score: {:.2f}".format(best_score))
print("Best parameters: {}".format(best_parameters))

Answer 4

There are a few ways of making this faster:

1. Decrease the CV value, as mentioned by @jncraton
2. Decrease the search space for the hyperparameters (test only a few parameters or decrease the ranges for parameters)

Additionally, you might consider using a more efficient way of hyperparameter searching by using hyperopt or nevergrad.

Answer 5

If speed is the only only issue then i have few suggestions that will definitely improve the algorithm run time by 5-10times(which i experienced), without compromising on any other input:

1) Increase the number of jobs submitted in parallel, use (n_jobs = -1) in the algorithm parameters. This will run the algo in parallel instead of series(and will cut down by time by 3 to 4 times. (chk the below code).

class sklearn.model_selection.GridSearchCV(estimator, param_grid, scoring=None, **n_jobs=None**, iid='deprecated', refit=True, cv=None, verbose=0, pre_dispatch='2*n_jobs', error_score=nan, return_train_score=False)

2) You can use RandomSearchCV in place of grid search. This also work on similar principal but must more optimized version(actually it randomly searches for optimum parameters unlike grid search that does it for all combinations). This will cut down algo run time by 4-5 folds again.

3) Combination of RandomSearchCV with n_jobs = -1, this will help to cutdown time by 8-10 times.

Please try it on your problem and feedback if it solved your problem or not ?