Can I run this job quicker for GridSearchCV?

Question

I am using GridSearchCV for optimising my predictions and its been 5 hours now that the process is running.

I am running a fairly large dataset and I am afraid I have not optimised the parameters enough.

df_train.describe():

         Unnamed: 0           col1           col2           col3           col4          col5
count  8.886500e+05  888650.000000  888650.000000  888650.000000  888650.000000  888650.000000
mean   5.130409e+05       2.636784       3.845549       4.105381       1.554918       1.221922
std    2.998785e+05       2.296243       1.366518       3.285802       1.375791       1.233717
min    4.000000e+00       1.010000       1.010000       1.010000       0.000000       0.000000
25%    2.484332e+05       1.660000       3.230000       2.390000       1.000000       0.000000
50%    5.233705e+05       2.110000       3.480000       3.210000       1.000000       1.000000
75%    7.692788e+05       2.740000       3.950000       4.670000       2.000000       2.000000
max    1.097490e+06      90.580000      43.420000      99.250000      22.000000      24.000000

df_test.describe():
         Unnamed: 0      col1        col2        col3        col4        col5
count  390.000000  390.000000  390.000000  390.000000         0.0         0.0
mean   194.500000    3.393359    4.016821    3.761385         NaN         NaN
std    112.727548    4.504227    1.720292    3.479109         NaN         NaN
min      0.000000    1.020000    2.320000    1.020000         NaN         NaN
25%     97.250000    1.792500    3.272500    2.220000         NaN         NaN
50%    194.500000    2.270000    3.555000    3.055000         NaN         NaN
75%    291.750000    3.172500    4.060000    4.217500         NaN         NaN
max    389.000000   50.000000   18.200000   51.000000         NaN         NaN

The way I am using GridSearchCV is as follows:

rf_h = RandomForestRegressor()
rf_a = RandomForestRegressor()

# Using GridSearch for Optimisation
param_grid = {

'n_estimators': [200, 700],

'max_features': ['auto', 'sqrt', 'log2']
}

rf_g_h = GridSearchCV(estimator=rf_h, param_grid=param_grid, cv=3, n_jobs=-1)
rf_g_a = GridSearchCV(estimator=rf_a, param_grid=param_grid, cv=3, n_jobs=-1)

# Fitting dataframe to prediction engine
rf_g_h.fit(X_h, y_h)
rf_g_a.fit(X_a, y_a)

My questions are:

1. How do I optimise GridSearchCV for multicore processing? I don't mind running the system all night long to get best_params_ but, somewhere it is being bottlenecked and I cannot seem to understand why?

1. Am I using GridSearchCV correctly? How would I go about choosing param_grid for the best parameters because the whole point of this exercise is to get best_params_
2. What could be an optimal start to exploring best parameters?

Without GrisSearchCV, and using defaults for RandomForestRegressor, the process completes in 10 minutes. I am running it for the first time on my dataset and I am curious of the results. I have a relatively capable computer running on Ryzen 7, 8 cores and a 32 GB RAM.

Edit: Corrected code as per suggestions. I saw a massive improvement from all night running with no output to improved output in 9 minutes

Answer 1

To extend my comment:

As I mentioned you can set the parameter n_jobs to -1 or instead using RandomizedGridSearch (which also receives n_jobs parameter)

Regarding to the parameter grid, I always select my grid so that the default values are included and from there, some values less and greater than the default (for continuous parameters) and the same logic for the categorical parameters.

Another heuristic would be to tune the parameters individually and using validation curves may help to find "adequate" parameters, so once you have found "the best" hyper parameters individually you can test the cv score using all those parameters that were found alone (and this might also give an idea of the range of parameters for which your model is performing well).

The later approach has of course drawbacks, like omitting the fact that a grid is multidimensional and the individual performance on one parameter may not be the "best" in combination with all the set of parameters.

Answer 2

Try Optuna which is relatively faster than GridSearchCV. Also n_jobs = -1 further reduces time. Another point is to tune parameters that matter. Not all parameters will give you maximum improvement in results. Read this blog for further info: https://blog.dataiku.com/narrowing-the-search-which-hyperparameters-really-matter