1
$\begingroup$

I am using a Random search for a light gradient boosting predicting a binary outcome with 33 features.

pipe = Pipeline(steps=[
                   ('scaler', MinMaxScaler()),                       
                   ('lgb',lgb.LGBMClassifier(verbose=-1))
                   
                  ]
           )
param_distributions = {                                        
                  "lgb__scale_pos_weight":[1.85],
                  "lgb__max_depth": [6],
                  "lgb__num_estimators": [100],
                  "lgb__learning_rate": [0.16],
                  "lgb__num_leaves" :[50,100],                                       
                  "lgb__objective":['binary'],                                              
                  "lgb__min_child_samples":[20,10,30],
                  "lgb__min_split_gain":[0.5,0.6,0.7,0.8,0.9,1],
                  "lgb__min_child_weight":[10],                      
                  "lgb__reg_alpha": [5,6],
                  "lgb__reg_lambda": [7,8],
                  "lgb__subsample":[0.5,0.6,0.7,0.8,0.9,1],
                  "lgb__subsample_freq":[5,10,15,20,25],
                  "lgb__colsample_bytree":[0.5,0.6,0.7,0.8,0.9,1],                                           
                  "lgb__importance_type" :['gain'],   
                  }# Set up score
 kfold = StratifiedKFold(n_splits=5, shuffle=True, random_state=42) 
 search = RandomizedSearchCV(estimator=pipe,
                        param_distributions=param_distributions,
                        cv=kfold, 
                        scoring ='f1',
                        refit=True, 
                        n_jobs=-1, 
                        verbose=1,
                       n_iter=1000,
                       return_train_score=True)
 np.random.seed(1234)
 grid_result=search.fit(X_train, y_train)

To check overfitting/underfitting, is np.nanmean() the right way to compare the f1 mean train score and f1 mean test score?

np.nanmean(search.cv_results_["mean_test_score"])
0.48542974980622244

np.nanmean(search.cv_results_["mean_train_score"])
0.6995323465268991

If it's correct (since the mean train score is greater than the mean test score), is the model overfitting?

$\endgroup$

1 Answer 1

1
$\begingroup$

What you have done so far

Given that you have specified RandomizedSearchCV(scoring="f1"), then we can assume that the values stored in mean_test_score and mean_train_score in the cv_results_ dictionary are the F1 scores across all the parameter combinations tried.

However, the values in mean_test_score and mean_train_score are for the total number of hyperparameter combinations that your X attempted, which is equal to n_iter=1000 attempts. This will include both poor combinations (that you probably don't want), good combinations of parameters, and one model that outperforms all the other models based on the metric you chose (F1 score).

This is based from the official found at scikit-learn.org.

Recommendations on what you need to do

The goal of grid search is to tune the model's hyperparameters (finding a combination of hyperparameters that work well for your problem). Therefore, to measure if your model is overfitting or not, you want to extract the best model that was found by exploring the search space you defined in the param_distributions dictionary. Then you can establish if that model is overfitting or not, rather than looking at all the models your grid search fitted.

Since you specified you want to know if your model is overfitting or not, you need to start by getting the best model from your whole grid search process. You can directly access it via the best_estimator_ attribute since you have the option refit=True. Then you can calculate some key metrics such as F1 score, to assess if there is a big difference between your sets and determine if your model is overfitting or not.

Example:

from sklearn.metrics import f1_score


clf_opt = search.best_estimator_

preds_train = clf_opt.predict(X_train, y_train)
preds_test = clf_opt.predict(X_test, y_test)

print(f"Train F1 score = {f1_score(y_train, preds_train)}")
print(f"Test F1 score = {f1_score(y_train, preds_train)}")

Note on usage of test set: I did notice you are using train/test sets. Remember that the test set should remain unseen by the model until the very end. You would usually use a third set, called validation set, to help fine-tune your model. In your grid search, you can use the train set to fit your model, then make predictions on the validation set. Once you've got your best model, you can measure the score between train/validation, and assess if it's overfitting or not. Only at the end should you use the test set.

$\endgroup$
2
  • $\begingroup$ Thank you for your response. About the validation set, I am using the StratifiedKFold cross-validation, so does it still need to use a validation set? $\endgroup$ Commented Sep 7, 2023 at 16:05
  • $\begingroup$ Hi @NimaYousefi, if you're using cross-validation you're good! $\endgroup$ Commented Sep 8, 2023 at 8:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.