1
$\begingroup$

I've created a MultinomialNB classifier model by which I'm trying to label some test texts:

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn import preprocessing
from sklearn.naive_bayes import MultinomialNB

tfv = TfidfVectorizer(strip_accents='unicode', analyzer='word',token_pattern=r'\w{1,}',
                      use_idf=1,smooth_idf=1,sublinear_tf=1)

# df['text'] is a long string text of words
tfv.fit(df['text'])

lbl_enc = preprocessing.LabelEncoder()

# df['which_subject'] is one of the following 7 subjects: ['Educational', 'Political', 'Sports', 'Tech', 'Social', 'Religions', 'Economics']
y = lbl_enc.fit_transform(df['which_subject'])

xtrain_tfv = tfv.transform(df['text'])

# xtest_tfv has 7 samples
xtest_tfv = tfv.transform(test_df['text'])

clf = MultinomialNB()
clf.fit(xtrain_tfv, y)

y_test_preds = clf.predict_proba(xtest_tfv)

Now y_test_preds is as follows:

enter image description here

0.255328    0.118111    0.129958    0.123368    0.119301    0.131098    0.122836
0.122814    0.265444    0.117637    0.13531     0.116697    0.122812    0.119286
0.131485    0.114459    0.258224    0.122414    0.118132    0.134005    0.12128
0.125075    0.131948    0.122668    0.258655    0.116518    0.119995    0.12514
0.124356    0.116987    0.121706    0.119796    0.266172    0.127231    0.123751
0.132295    0.1192      0.13366     0.119445    0.123186    0.257318    0.114895
0.126779    0.118406    0.123723    0.127393    0.122539    0.117509    0.263652

As you see, all of the elements are less than 0.5. Does this table show anything? Can I conclude that the classifier is not able to label test text?

$\endgroup$
1
$\begingroup$

No, your classifier can label text. It doesn't do it well but it is still almost 2 times better than random (for 7 classes, random will get you ~0.15 accuracy).

Looking at the test set is not enough. You need to create the same confusion matrix for you training set.

If the results you will get for the test set are similar in magnitude than maybe your model is too simple for the task or maybe you haven't trained it long enough.

If the results of the test set are good, than you might have a generalization problem (overfitting), which means that you need to increase the regularization during training. It also might mean that your training set comes from a different distribution than your test set.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks. Some questions from your answer: I'd be be glad if you could answer some of those, since the answer seems very informative. 1. Could you please clarify why a random choice is 0.15? 2. By saying are similar in magnitude you mean if I'm getting rather similar confusion matrix ( for example : maximum predict_proba of 0.25 for each column) on train set? 3. What do you mean by saying maybe your model is too simple ? How can I turn it into a more advanced one? 4. How can I increase regularization? $\endgroup$ – hyTuev Jan 20 '19 at 10:37
  • $\begingroup$ 1. Random will just be a uniform distribution over all classes, so your accuracy will be 1/7 which is about 0.15 (number of classes). 2. Yes, that's what I meant. 3. You are using a Naive Bayes classifier, which is a very simple model, with good but limited abilities. You can always try more complex classifier types like SVMs or Neural Networks. 4. I was speaking in general terms about classifiers (I'm not sure about regularization in Naive Bayes models) $\endgroup$ – Mark.F Jan 20 '19 at 11:54
  • $\begingroup$ Oh, what a mistake. Of course a random choice is 1/7. Thanks again for the answer and comments. $\endgroup$ – hyTuev Jan 20 '19 at 13:00
  • $\begingroup$ Could you please look at my new question that is directly related to this question? datascience.stackexchange.com/questions/44306/… $\endgroup$ – hyTuev Jan 20 '19 at 20:33
  • $\begingroup$ I see that someone has already beat me to it. $\endgroup$ – Mark.F Jan 22 '19 at 13:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.