I have some doubts regarding the approach to build a classifier such as Multinomial Naive Bayes or SVM. I will go through the steps to see if the approach is fine. I have not a lot of experience in model build, so any suggestions would be great! My dataset has approx. 1115 obs having positive value (0) and 66 obs having negative value (1). The distribution of the dependant variable is shown in the figure below.enter image description here

I split the dataset into train (70) and test (30), using stratify (it should help in case of such discrepancy between classes, hopefully):

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.30, stratify=y)

Then I imported the sim model to create a SVM classifier:

from sklearn import svm

clf = svm.SVC(kernel='linear') 

clf.fit(X_train, y_train)

And to Predict the response for test dataset I used the following:

y_pred = clf.predict(X_test)

For accuracy calculation, I used the following code:

from sklearn import metrics

print("Accuracy:",metrics.accuracy_score(y_test, y_pred))

print("Precision:",metrics.precision_score(y_test, y_pred))

print("Recall:",metrics.recall_score(y_test, y_pred))

getting different values every time I re-run it:

Accuracy: 0.75

                precision    recall  f1-score   support

           0       0.95      0.79      0.86       316
           1       0.08      0.30      0.13        20

    accuracy                           0.76       336
   macro avg       0.51      0.54      0.49       336
weighted avg       0.90      0.76      0.82       336

2nd re-run

Accuracy:  0.8005952380952381

                precision    recall  f1-score   support

           0       0.94      0.84      0.89       316
           1       0.07      0.20      0.11        20

    accuracy                           0.80       336
   macro avg       0.51      0.52      0.50       336
weighted avg       0.89      0.80      0.84       336

Confusion Matrix: 
 [[265  51]
 [ 16   4]]

3rd re-run

Accuracy:  0.7797619047619048

                precision    recall  f1-score   support

           0       0.94      0.81      0.87       316
           1       0.08      0.25      0.12        20

    accuracy                           0.78       336
   macro avg       0.51      0.53      0.50       336
weighted avg       0.89      0.78      0.83       336

Confusion Matrix: 
 [[257  59]
 [ 15   5]]

I have a couple of questions on these results and I hope to find answers to them:

  • Which value should I take into account for saying that my model has accuracy of ...?
  • Does it make sense to run a model where there are so a few values = 1 for the dependent variable?

Which value should I take into account for saying that my model has accuracy of ...?

None. Accuracy is practically meaningless in such class imbalanced settings; the metrics of interest here are precision, recall, and f1 score.

Now, it's true that the values of these metrics also fluctuate between runs, much similarly with the reported accuracy. But this is to be expected due to the small-sample effects - your sample is so small that even a difference in the classification of a couple of samples in your (even smaller) validation set is enough to give the observed discrepancies.

Does it make sense to run a model where there are so a few values = 1 for the dependent variable?

Indeed it does; there are plenty of applications where the positive values are a very small percentage of the whole dataset (thing of sick people versus the general population of healthy ones, or engine faults versus long running times where no fault is present). That's why this is a (large enough) sub-topic of machine learning called class imbalance or imbalanced classification, with its own specific approaches. I suggest you start googling ruthlessly...

  • $\begingroup$ thanks desertnaut. May I ask you what you think about the results of precision, recall and f1 score? I would like to understand why there is an acceptable value in accuracy but not there... $\endgroup$ – Val Oct 13 '20 at 22:05
  • 1
    $\begingroup$ @Val this always depends on the specific business problem you are trying to solve; in settings like the examples I have mentioned, and if your 1 (minority) class is the actual class of interest (e.g. sick people, faulty engines etc), f1 scores in the range of 11-13% are really poor. $\endgroup$ – desertnaut Oct 13 '20 at 22:07
  • $\begingroup$ I am trying to identify fake news mainly, so 1 represents my actual class of interest. I followed all the rules of pre-processing and did all my best to remove words that could be included in both sets trying to improve the model and results. So I am wondering if it does not depend on the dataset and the number of 1 (very low) there $\endgroup$ – Val Oct 13 '20 at 22:10
  • $\begingroup$ @Val as already said, start googling ruthlessly; class imbalance is a whole sub-world with dedicated approaches, tools, and techniques designed for this kind of problems. $\endgroup$ – desertnaut Oct 13 '20 at 22:12
  • $\begingroup$ thanks @desertnaut. I will do it $\endgroup$ – Val Oct 13 '20 at 22:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.