Classification Threshold Tuning with GridSearchCV

Question

In Scikit-learn, GridSearchCV can be used to validate a model against a grid of parameters. A short example for grid-search cv against some of DecisionTreeClassifier parameters is given as follows:

model = DecisionTreeClassifier()
params = [{'criterion':["gini","entropy"],"max_depth":[1,2,3,4,5,6,7,8,9,10],"class_weight":["balanced"]}]
GSCV = GridSearchCV(model,params,scoring="f1_micro")
GSCV.fit(X_train,y_train)
GSCV.best_params_

Now, I am only concerned with binary classification. It is the case for many algorithms that they compute a probability score, and set the decision threshold at 0.5. My question is the following: If I want to consider the decision threshold as another parameter of the grid search (along with the existing parameters), is there a standard way to do this with GridSearchCV? For instance, something like the last parameter "decision_threshold" in the following tunegrid would be ideal:

params = [{'criterion':["gini","entropy"],"max_depth":[1,2,3,4,5,6,7,8,9,10],"class_weight":["balanced"], "decision_threshold": [0.1,0.2,...,0.9]}]

Needless to say, I am not interested in a particular solution working only for DecisionTreeClassifier. Instead, a general solution for any classifier that uses a probability decision threshold that can be tuned. Preferably, I would like to keep my current GridSearchCV if possible.

Answer 1

As far as I know, you cannot add the model's threshold as a hyperparameter but to find the optimal threshold you can do as follows:

1. make a the standard GridSearchCV but use the roc_auc as metric as per step 2

    model = DecisionTreeClassifier()
    params = [{'criterion':["gini","entropy"],"max_depth":[1,2,3,4,5,6,7,8,9,10],"class_weight":["balanced"]}]
    GSCV = GridSearchCV(model,params,scoring="roc_auc")
    GSCV.fit(X_train,y_train)
    GSCV.best_params_
    best_model = GSCV.best_estimator_
   

2. Once you have the best hyper parameters set you can obtain the threshold that maximizes the roc curve as follows:

    from sklearn.metrics import roc_curve
    preds = best_model.predict_proba(X_train)[:,1]
   
    fpr, tpr, thresholds = roc_curve(y_train, preds)
    optimal_idx = np.argmax(tpr - fpr)
    optimal_threshold = thresholds[optimal_idx]
   

This threshold will give you the lowest false positive rate and the highest true positive rate