Classification Threshold Tuning with GridSearchCV

In Scikit-learn, GridSearchCV can be used to validate a model against a grid of parameters. A short example for grid-search cv against some of DecisionTreeClassifier parameters is given as follows:

model = DecisionTreeClassifier()
params = [{'criterion':["gini","entropy"],"max_depth":[1,2,3,4,5,6,7,8,9,10],"class_weight":["balanced"]}]
GSCV = GridSearchCV(model,params,scoring="f1_micro")
GSCV.fit(X_train,y_train)
GSCV.best_params_


Now, I am only concerned with binary classification. It is the case for many algorithms that they compute a probability score, and set the decision threshold at 0.5. My question is the following: If I want to consider the decision threshold as another parameter of the grid search (along with the existing parameters), is there a standard way to do this with GridSearchCV? For instance, something like the last parameter "decision_threshold" in the following tunegrid would be ideal:

params = [{'criterion':["gini","entropy"],"max_depth":[1,2,3,4,5,6,7,8,9,10],"class_weight":["balanced"], "decision_threshold": [0.1,0.2,...,0.9]}]


Needless to say, I am not interested in a particular solution working only for DecisionTreeClassifier. Instead, a general solution for any classifier that uses a probability decision threshold that can be tuned. Preferably, I would like to keep my current GridSearchCV if possible.

As far as I know you cannot add the model's threshold as hyperparameter but in order to find the optimal threshold you can do as follows:

1. make a the standard GridSearchCV but use the roc_auc as metric as per step 2

 model = DecisionTreeClassifier()
params = [{'criterion':["gini","entropy"],"max_depth":[1,2,3,4,5,6,7,8,9,10],"class_weight":["balanced"]}]
GSCV = GridSearchCV(model,params,scoring="roc_auc")
GSCV.fit(X_train,y_train)
GSCV.best_params_
best_model = GSCV.best_estimator_

2. Once you have the best hyper parameters set you can obtain the threshold that maximizes the roc curve as follows:

 from sklearn.metrics import roc_curve
preds = best_model.predict_proba(X_train)[:,1]

fpr, tpr, thresholds = roc_curve(y_train, preds)
optimal_idx = np.argmax(tpr - fpr)
optimal_threshold = thresholds[optimal_idx]


This threshold will give you the lowest false positive rate and the highest true positive rate