2
$\begingroup$

In Scikit-learn, GridSearchCV can be used to validate a model against a grid of parameters. A short example for grid-search cv against some of DecisionTreeClassifier parameters is given as follows:

model = DecisionTreeClassifier()
params = [{'criterion':["gini","entropy"],"max_depth":[1,2,3,4,5,6,7,8,9,10],"class_weight":["balanced"]}]
GSCV = GridSearchCV(model,params,scoring="f1_micro")
GSCV.fit(X_train,y_train)
GSCV.best_params_

Now, I am only concerned with binary classification. It is the case for many algorithms that they compute a probability score, and set the decision threshold at 0.5. My question is the following: If I want to consider the decision threshold as another parameter of the grid search (along with the existing parameters), is there a standard way to do this with GridSearchCV? For instance, something like the last parameter "decision_threshold" in the following tunegrid would be ideal:

params = [{'criterion':["gini","entropy"],"max_depth":[1,2,3,4,5,6,7,8,9,10],"class_weight":["balanced"], "decision_threshold": [0.1,0.2,...,0.9]}]

Needless to say, I am not interested in a particular solution working only for DecisionTreeClassifier. Instead, a general solution for any classifier that uses a probability decision threshold that can be tuned. Preferably, I would like to keep my current GridSearchCV if possible.

$\endgroup$

1 Answer 1

3
$\begingroup$

As far as I know, you cannot add the model's threshold as a hyperparameter but to find the optimal threshold you can do as follows:

  1. make a the standard GridSearchCV but use the roc_auc as metric as per step 2

     model = DecisionTreeClassifier()
     params = [{'criterion':["gini","entropy"],"max_depth":[1,2,3,4,5,6,7,8,9,10],"class_weight":["balanced"]}]
     GSCV = GridSearchCV(model,params,scoring="roc_auc")
     GSCV.fit(X_train,y_train)
     GSCV.best_params_
     best_model = GSCV.best_estimator_
    
  2. Once you have the best hyper parameters set you can obtain the threshold that maximizes the roc curve as follows:

     from sklearn.metrics import roc_curve
     preds = best_model.predict_proba(X_train)[:,1]
    
     fpr, tpr, thresholds = roc_curve(y_train, preds)
     optimal_idx = np.argmax(tpr - fpr)
     optimal_threshold = thresholds[optimal_idx]
    

This threshold will give you the lowest false positive rate and the highest true positive rate

$\endgroup$
2
  • $\begingroup$ Isn't it a problem that the data for the predictions has been used to train? $\endgroup$
    – VeloFX
    Sep 30, 2022 at 22:00
  • $\begingroup$ @VeloFX not because the prediction has only used train data, never test. $\endgroup$
    – skan
    Aug 2 at 1:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.