1
$\begingroup$

I've created a MultinomialNB classifier model by which I'm trying to label some test texts:

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn import preprocessing
from sklearn.naive_bayes import MultinomialNB

tfv = TfidfVectorizer(strip_accents='unicode', analyzer='word',token_pattern=r'\w{1,}',
                      use_idf=1,smooth_idf=1,sublinear_tf=1)

# df['text'] is a long string text of words
tfv.fit(df['text'])

lbl_enc = preprocessing.LabelEncoder()

# df['which_subject'] is one of the following 7 subjects: ['Educational', 'Political', 'Sports', 'Tech', 'Social', 'Religions', 'Economics']
y = lbl_enc.fit_transform(df['which_subject'])

xtrain_tfv = tfv.transform(df['text'])

# xtest_tfv has 7 samples
xtest_tfv = tfv.transform(test_df['text'])

clf = MultinomialNB()
clf.fit(xtrain_tfv, y)

y_test_preds = clf.predict_proba(xtest_tfv)

Now y_test_preds is as follows:

enter image description here

0.255328    0.118111    0.129958    0.123368    0.119301    0.131098    0.122836
0.122814    0.265444    0.117637    0.13531     0.116697    0.122812    0.119286
0.131485    0.114459    0.258224    0.122414    0.118132    0.134005    0.12128
0.125075    0.131948    0.122668    0.258655    0.116518    0.119995    0.12514
0.124356    0.116987    0.121706    0.119796    0.266172    0.127231    0.123751
0.132295    0.1192      0.13366     0.119445    0.123186    0.257318    0.114895
0.126779    0.118406    0.123723    0.127393    0.122539    0.117509    0.263652

As you see, all of the elements are less than 0.5. Does this table show anything? Can I conclude that the classifier is not able to label test text?

$\endgroup$

1 Answer 1

1
$\begingroup$

No, your classifier can label text. It doesn't do it well but it is still almost 2 times better than random (for 7 classes, random will get you ~0.15 accuracy).

Looking at the test set is not enough. You need to create the same confusion matrix for you training set.

If the results you will get for the test set are similar in magnitude than maybe your model is too simple for the task or maybe you haven't trained it long enough.

If the results of the test set are good, than you might have a generalization problem (overfitting), which means that you need to increase the regularization during training. It also might mean that your training set comes from a different distribution than your test set.

$\endgroup$
6
  • $\begingroup$ Thanks. Some questions from your answer: I'd be be glad if you could answer some of those, since the answer seems very informative. 1. Could you please clarify why a random choice is 0.15? 2. By saying are similar in magnitude you mean if I'm getting rather similar confusion matrix ( for example : maximum predict_proba of 0.25 for each column) on train set? 3. What do you mean by saying maybe your model is too simple ? How can I turn it into a more advanced one? 4. How can I increase regularization? $\endgroup$
    – hyTuev
    Jan 20, 2019 at 10:37
  • $\begingroup$ 1. Random will just be a uniform distribution over all classes, so your accuracy will be 1/7 which is about 0.15 (number of classes). 2. Yes, that's what I meant. 3. You are using a Naive Bayes classifier, which is a very simple model, with good but limited abilities. You can always try more complex classifier types like SVMs or Neural Networks. 4. I was speaking in general terms about classifiers (I'm not sure about regularization in Naive Bayes models) $\endgroup$
    – Mark.F
    Jan 20, 2019 at 11:54
  • $\begingroup$ Oh, what a mistake. Of course a random choice is 1/7. Thanks again for the answer and comments. $\endgroup$
    – hyTuev
    Jan 20, 2019 at 13:00
  • $\begingroup$ Could you please look at my new question that is directly related to this question? datascience.stackexchange.com/questions/44306/… $\endgroup$
    – hyTuev
    Jan 20, 2019 at 20:33
  • $\begingroup$ I see that someone has already beat me to it. $\endgroup$
    – Mark.F
    Jan 22, 2019 at 13:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.