Actually I have two questions. One of them is related the bug of sklearn SVM model and the other one is about ROC-AUC score.
- My first question is related to ROC-AUC score but also includes a bug similar to that (https://github.com/scikit-learn/scikit-learn/issues/13662) and my sklearn versiton is: scikit_learn-0.24.2-cp36-cp36m-macosx_10_13_x86_64.whl
I have imbalanced data which have 4672 positive and 14459 negative samples. I trained SVM model by following that code:
kernel='rbf' c = 0.01 Gamma = 0.5 svm_model = SVC(kernel=kernel,C=c, gamma= Gamma, probability=True) ss = svm_model.fit(X_train,y_train) y_true_SVM1, y_pred_SVM1 = y_test, ss.predict(X_test) y_true_SVM,yy_pred_SVM = flat(y_test), ss.predict_proba(X_test) #flat like a converting one hot representation
When I look at y_pred_SVM1 which is the output of predict function, I saw that all samples were predicted as negative. It is normal because of the imbalance. However, when I investigate yy_pred_SVM variable which keeps the predicted probabilities I saw that there are 6 samples which were predicted as positives:
[3.0000009e-14 1.0000000e+00] [0.39734363 0.60265637] [0.04542387 0.95457613] [0.04544127 0.95455873] [0.30997869 0.69002131] [4.13349563e-09 9.99999996e-01]
I thought that it is a bug but I couldn't be sure. Because If there is a such bug how can we sure about the other probabilities. In their github page such a bug was reported for cross validation but I didn't use CV.
Moreover, I tried to evaluate my results with respect to some metrics:
SVM Accuracy score on our Test data: 0.7558795860771401 SVM Sensitivity score on our Test data: 0.0 SVM Specificity score on our Test data: 1.0 SVM MCC score on our Test data: 0.0 SVM ROC-AUC score on our Test data: 0.8524902491867827 SVM Average Precision Score score on our Test data: 0.7477108938150628 SVM F1-Score on our Test data: 0.0
However, I couldn't figure it out that even all classes were classified as negatives how can ROC-AUC score will be 0.85. I used this code for that:
rocAuc = metrics.roc_auc_score(y_true_SVM[:,1], yy_pred_SVM[:,1])
I know that there can be a threshold different from 0.5 and it could create such a case. Because of that reason I plotted the probs of positive and negative classes:
As you can see on the image above, there is not a distinct threshold for classes. Also we can understand from that prob array:
array([[0.63341647, 0.36658353], [0.7541376 , 0.2458624 ], [0.75364027, 0.24635973], ..., [0.75417909, 0.24582091], [0.91853779, 0.08146221], [0.75418213, 0.24581787]])
probabilities for being a positive were gathered around 2.45.
How should I approach that results and how should I fix that inconsistency between pred() and predict_proba() functions ?
By the way when I use class_weight
svm_model2 = SVC(kernel=kernel,C=c, gamma= Gamma,class_weight='balanced',probability=True)
pred() and predict_proba() gives same results in reverse order like in the link above. But this time my result became:
SVM Accuracy score on our Test data: 0.24412041392285982 SVM Sensitivity score on our Test data: 1.0 SVM Specificity score on our Test data: 0.0 SVM MCC score on our Test data: 0.0 SVM ROC-AUC score on our Test data: 0.8521257552159206 SVM Average Precision Score score on our Test data: 0.7472358224145087 SVM F1-Score on our Test data: 0.3924385633270321
Thank you for your help!