How to do grid search for Catboost with categorical_cols

Question

I know it's easy to do grid search for a simple Catboost model, such as in here: https://medium.com/aiplusoau/hyperparameter-tuning-a5fe69d2a6c7

by running something like

cbc = CatBoostRegressor()

#create the grid
grid = {'max_depth': [3,4,5],'n_estimators':[100, 200, 300]}

#Instantiate GridSearchCV
gscv = GridSearchCV (estimator = cbc, param_grid = grid, scoring 
='accuracy', cv = 5)

#fit the model
gscv.fit(X,y)

#returns the estimator with the best performance
print(gscv.best_estimator_)

Method like this did not have the input of categorical columns in the Catboost model.

But my question is how can I do grid search for categorical_cols specified?

For example, here is my code how I assign the categorical columns:

categorical_cols =  ['site_number','product_key', 'manufacturer_desc']


# initialize Pool
train_pool = Pool(X_train, 
                  y_train, 
                  cat_features=categorical_cols)
test_pool = Pool(X_test, 
                 cat_features=categorical_cols) 


# specify the training parameters 
model = CatBoostRegressor(iterations=150, 
                          learning_rate = 0.5,
                          depth=8,  
                          random_seed = 42
                          )
#train the model
model.fit(train_pool)

But this is the model without grid search. The question is how can I still do the grid search with the above categorical_cols specified. The train_pool and test_pool is already specified, not sure what's a best way.

Thanks!

Answer 1

To perform a grid search with specified categorical columns in CatBoost, you can use the GridSearchCV function from Scikit-learn. You can define a parameter grid with different values for the hyperparameters you want to tune, including the categorical columns. Here's an example:

from catboost import Pool, CatBoostRegressor
from sklearn.model_selection import GridSearchCV

# define the parameter grid
params = {
    'iterations': [100, 150, 200],
    'learning_rate': [0.1, 0.5, 1],
    'depth': [6, 8, 10],
    'cat_features': [['site_number', 'product_key', 'manufacturer_desc'],
                     ['site_number', 'product_key'],
                     ['product_key', 'manufacturer_desc']]
}

# initialize Pool
train_pool = Pool(X_train, 
                  y_train, 
                  cat_features=categorical_cols)
test_pool = Pool(X_test, 
                 cat_features=categorical_cols) 

# initialize the model
cat = CatBoostRegressor(random_seed=42, silent=True)

# perform grid search with 5-fold cross-validation
grid_search = GridSearchCV(cat, param_grid=params, cv=5)

# fit the grid search to the data
grid_search.fit(train_pool)

# print the best hyperparameters
print(grid_search.best_params_)

In this example, the params dictionary contains different values for the iterations, learning_rate, depth, and cat_features hyperparameters. The cat_features parameter takes a list of lists, where each list is a different combination of categorical columns. The GridSearchCV function performs a grid search with 5-fold cross-validation to find the best combination of hyperparameters. The best hyperparameters can be accessed with the best_params_ attribute of the GridSearchCV object (including the ones you provided in your code if they merit the best results).