2
$\begingroup$

I have chosen threshold value with below code to get 90% precision classifier

from sklearn.model_selection import cross_val_predict
y_train_pred = cross_val_predict(sgd_clf, X_train, y_train, cv=3)

z_scores = cross_val_predict(sgd_clf, X_train, y_train, method='decision_function')

from sklearn.metrics import precision_recall_curve
precisions, recalls, thresholds = precision_recall_curve(y_train_pred, z_scores)

threshold_90_precision = thresholds[np.argmax(precisions >= 0.9)]

y_train_pred_90percent_precision = (z_scores >= threshold_90_precision)
print(precision_score(y_train, y_train_pred_90percent_precision))

I'm expecting precision_score to be 90% but it returned 95%. Is this expected? Anything incorrect with my code? If it's expected, can you please explain the reason?

$\endgroup$
2
  • $\begingroup$ Isn’t 95% precision better than 90%? $\endgroup$
    – Dave
    Jun 28 '20 at 6:37
  • $\begingroup$ 95% is better than 90% but I have trained my classifier to be 90% classifier, and expect precision score as 90% $\endgroup$
    – NewDev
    Jun 28 '20 at 18:22
2
$\begingroup$
threshold_90_precision = thresholds[np.argmax(precisions >= 0.9)]

Above snippet is not doing what you are expecting it to do.

Try these changes

precisions[precisions < 0.9] = 1
threshold_90_precision = thresholds[np.argmin(precisions)]

Also, I am not sure whether you are calculating the accuracy properly since z_scores is decision function, not Class.

This is a working example using method='predict_proba' for 40%, you may change to 90%

model.fit(x_train, y_train)

from sklearn.model_selection import cross_val_predict
y_train_pred = cross_val_predict(model, x_train, y_train, cv=3)

z_scores = cross_val_predict(model, x_train, y_train, method='predict_proba')

from sklearn.metrics import precision_recall_curve
precisions, recalls, thresholds = precision_recall_curve(y_train_pred, z_scores[:,0])

import numpy as np
precisions[precisions < 0.4] = 1
threshold_90_precision = thresholds[np.argmin(precisions)]

y_train_pred_90percent_precision = z_scores[:,0] >= threshold_90_precision
from sklearn.metrics import precision_score
print(precision_score(y_train, y_train_pred_90percent_precision))
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.