0
$\begingroup$

Actually I have two questions. One of them is related the bug of sklearn SVM model and the other one is about ROC-AUC score.

  1. My first question is related to ROC-AUC score but also includes a bug similar to that (https://github.com/scikit-learn/scikit-learn/issues/13662) and my sklearn versiton is: scikit_learn-0.24.2-cp36-cp36m-macosx_10_13_x86_64.whl

I have imbalanced data which have 4672 positive and 14459 negative samples. I trained SVM model by following that code:

kernel='rbf'
c = 0.01
Gamma = 0.5
svm_model = SVC(kernel=kernel,C=c, gamma= Gamma, probability=True)
ss = svm_model.fit(X_train,y_train)
y_true_SVM1, y_pred_SVM1 = y_test, ss.predict(X_test) 
y_true_SVM,yy_pred_SVM = flat(y_test), ss.predict_proba(X_test) #flat like a converting one hot representation

When I look at y_pred_SVM1 which is the output of predict function, I saw that all samples were predicted as negative. It is normal because of the imbalance. However, when I investigate yy_pred_SVM variable which keeps the predicted probabilities I saw that there are 6 samples which were predicted as positives:

[3.0000009e-14 1.0000000e+00]
[0.39734363 0.60265637]
[0.04542387 0.95457613]
[0.04544127 0.95455873]
[0.30997869 0.69002131]
[4.13349563e-09 9.99999996e-01]

I thought that it is a bug but I couldn't be sure. Because If there is a such bug how can we sure about the other probabilities. In their github page such a bug was reported for cross validation but I didn't use CV.

Moreover, I tried to evaluate my results with respect to some metrics:

SVM Accuracy score on our Test data: 0.7558795860771401
SVM Sensitivity score on our Test data: 0.0
SVM Specificity score on our Test data: 1.0
SVM MCC score on our Test data: 0.0
SVM ROC-AUC score on our Test data: 0.8524902491867827
SVM Average Precision Score score on our Test data: 0.7477108938150628
SVM F1-Score on our Test data: 0.0

However, I couldn't figure it out that even all classes were classified as negatives how can ROC-AUC score will be 0.85. I used this code for that:

rocAuc = metrics.roc_auc_score(y_true_SVM[:,1], yy_pred_SVM[:,1])

I know that there can be a threshold different from 0.5 and it could create such a case. Because of that reason I plotted the probs of positive and negative classes:

enter image description here

As you can see on the image above, there is not a distinct threshold for classes. Also we can understand from that prob array:

array([[0.63341647, 0.36658353],
       [0.7541376 , 0.2458624 ],
       [0.75364027, 0.24635973],
       ...,
       [0.75417909, 0.24582091],
       [0.91853779, 0.08146221],
       [0.75418213, 0.24581787]])

probabilities for being a positive were gathered around 2.45.

How should I approach that results and how should I fix that inconsistency between pred() and predict_proba() functions ?

By the way when I use class_weight

svm_model2 = SVC(kernel=kernel,C=c, gamma= Gamma,class_weight='balanced',probability=True)

pred() and predict_proba() gives same results in reverse order like in the link above. But this time my result became:

SVM Accuracy score on our Test data: 0.24412041392285982
SVM Sensitivity score on our Test data: 1.0
SVM Specificity score on our Test data: 0.0
SVM MCC score on our Test data: 0.0
SVM ROC-AUC score on our Test data: 0.8521257552159206
SVM Average Precision Score score on our Test data: 0.7472358224145087
SVM F1-Score on our Test data: 0.3924385633270321

Thank you for your help!

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.