2
$\begingroup$

I am searching for best hyper parameters of XGBRegressor using GridSearchCV. Here is the code:

from xgboost import XGBRegressor
from sklearn.model_selection import GridSearchCV, KFold

param_grid_xgb_b1 = {
    'num_leaves': np.arange(20, 500, 5),
    'max_depth': np.arange(5, 50, 2),
    'reg_alpha': np.arange(0.01, 0.5, 0.01),
    'min_data_in_leaf': np.arange(50, 1_000, 10),
    'colsample_bytree': [0.65, 0.75, 0.85, 0.95, 1],
    'subsample': [0.65, 0.75, 0.85, 0.95, 1]
    }
kfold_xgb_b1 = KFold(n_splits=5, shuffle=True, random_state=42).split(X=X_train, y=y_train)
grid_search_xgb_b1 = GridSearchCV(estimator=XGBRegressor(learning_rate=0.01, n_estimators=20_000, random_state=0,
                                                         tree_method='gpu_hist', subsample_freq=5),
                                  param_grid=param_grid_xgb_b1,
                                  scoring=rmse_score, cv=kfold_xgb_b1)
grid_search_xgb_b1.fit(X=X_train, y=y_train, verbose=3)

I am using Google Colab with GPU. Machine is dual core CPU with 12.72GB of RAM. And GPU is Tesla T4 with CUDA Version 10.1.

How do I prevent crash and search for hyper parameters?

EDIT 1:
Crash type: RAM filling up;
size of data: 300,000 rows and 40 columns.

$\endgroup$
7
2
$\begingroup$

I assume its crashing because not enough RAM. So I also assume your data is quite big. Your search grid is quite big. So this will definitely take some time.

In order to speed up the training and overloading the RAM. You can fit the model in a subsample of data. Theoretically if your data is big enough and you sample it, when you use the whole model the optimal hyperparameters should be quite similar.

Once you do, see what is the actual improvement of changing hyperparameters, hyperparameters in xgboost shouldn't be sooooo important. They will make a change but not a huge change.

So my suggestion will be to sample your data by 1% or something similar to search for hyperparameters and then fitting the model in the whole training set.

And not search in such a huge hyperparameter space.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.