Model accuracy: how to determine it?

Question

I have some doubts regarding the approach to build a classifier such as Multinomial Naive Bayes or SVM. I will go through the steps to see if the approach is fine. I have not a lot of experience in model build, so any suggestions would be great! My dataset has approx. 1115 obs having positive value (0) and 66 obs having negative value (1). The distribution of the dependant variable is shown in the figure below.

I split the dataset into train (70) and test (30), using stratify (it should help in case of such discrepancy between classes, hopefully):

from sklearn.model_selection import train_test_split

y=df['Label']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.30, stratify=y)

Then I imported the sim model to create a SVM classifier:

from sklearn import svm

clf = svm.SVC(kernel='linear') 

clf.fit(X_train, y_train)

And to Predict the response for test dataset I used the following:

y_pred = clf.predict(X_test)

For accuracy calculation, I used the following code:

from sklearn import metrics

print("Accuracy:",metrics.accuracy_score(y_test, y_pred))

print("Precision:",metrics.precision_score(y_test, y_pred))

print("Recall:",metrics.recall_score(y_test, y_pred))

getting different values every time I re-run it:

Accuracy: 0.75

                precision    recall  f1-score   support

           0       0.95      0.79      0.86       316
           1       0.08      0.30      0.13        20

    accuracy                           0.76       336
   macro avg       0.51      0.54      0.49       336
weighted avg       0.90      0.76      0.82       336

2nd re-run

Accuracy:  0.8005952380952381

                precision    recall  f1-score   support

           0       0.94      0.84      0.89       316
           1       0.07      0.20      0.11        20

    accuracy                           0.80       336
   macro avg       0.51      0.52      0.50       336
weighted avg       0.89      0.80      0.84       336

Confusion Matrix: 
 [[265  51]
 [ 16   4]]

3rd re-run

Accuracy:  0.7797619047619048

                precision    recall  f1-score   support

           0       0.94      0.81      0.87       316
           1       0.08      0.25      0.12        20

    accuracy                           0.78       336
   macro avg       0.51      0.53      0.50       336
weighted avg       0.89      0.78      0.83       336

Confusion Matrix: 
 [[257  59]
 [ 15   5]]

I have a couple of questions on these results and I hope to find answers to them:

• Which value should I take into account for saying that my model has accuracy of ...?
• Does it make sense to run a model where there are so a few values = 1 for the dependent variable?

Answer 1

Which value should I take into account for saying that my model has accuracy of ...?

None. Accuracy is practically meaningless in such class imbalanced settings; the metrics of interest here are precision, recall, and f1 score.

Now, it's true that the values of these metrics also fluctuate between runs, much similarly with the reported accuracy. But this is to be expected due to the small-sample effects - your sample is so small that even a difference in the classification of a couple of samples in your (even smaller) validation set is enough to give the observed discrepancies.

Does it make sense to run a model where there are so a few values = 1 for the dependent variable?

Indeed it does; there are plenty of applications where the positive values are a very small percentage of the whole dataset (thing of sick people versus the general population of healthy ones, or engine faults versus long running times where no fault is present). That's why this is a (large enough) sub-topic of machine learning called class imbalance or imbalanced classification, with its own specific approaches. I suggest you start googling ruthlessly...