I have some doubts regarding the approach to build a classifier such as Multinomial Naive Bayes or SVM. I will go through the steps to see if the approach is fine. I have not a lot of experience in model build, so any suggestions would be great!
My dataset has approx. 1115 obs having positive value (0) and 66 obs having negative value (1).
The distribution of the dependant variable is shown in the figure below.
I split the dataset into train (70) and test (30), using stratify (it should help in case of such discrepancy between classes, hopefully):
from sklearn.model_selection import train_test_split
y=df['Label']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.30, stratify=y)
Then I imported the sim model to create a SVM classifier:
from sklearn import svm
clf = svm.SVC(kernel='linear')
clf.fit(X_train, y_train)
And to Predict the response for test dataset I used the following:
y_pred = clf.predict(X_test)
For accuracy calculation, I used the following code:
from sklearn import metrics
print("Accuracy:",metrics.accuracy_score(y_test, y_pred))
print("Precision:",metrics.precision_score(y_test, y_pred))
print("Recall:",metrics.recall_score(y_test, y_pred))
getting different values every time I re-run it:
Accuracy: 0.75
precision recall f1-score support
0 0.95 0.79 0.86 316
1 0.08 0.30 0.13 20
accuracy 0.76 336
macro avg 0.51 0.54 0.49 336
weighted avg 0.90 0.76 0.82 336
2nd re-run
Accuracy: 0.8005952380952381
precision recall f1-score support
0 0.94 0.84 0.89 316
1 0.07 0.20 0.11 20
accuracy 0.80 336
macro avg 0.51 0.52 0.50 336
weighted avg 0.89 0.80 0.84 336
Confusion Matrix:
[[265 51]
[ 16 4]]
3rd re-run
Accuracy: 0.7797619047619048
precision recall f1-score support
0 0.94 0.81 0.87 316
1 0.08 0.25 0.12 20
accuracy 0.78 336
macro avg 0.51 0.53 0.50 336
weighted avg 0.89 0.78 0.83 336
Confusion Matrix:
[[257 59]
[ 15 5]]
I have a couple of questions on these results and I hope to find answers to them:
- Which value should I take into account for saying that my model has accuracy of ...?
- Does it make sense to run a model where there are so a few values = 1 for the dependent variable?