I want to use the Naive Bayes model as a baseline in an classification task that I am working. I found this really useful tutorial: https://www.geeksforgeeks.org/applying-multinomial-naive-bayes-to-nlp-problems/ and I want to apply it into my problem.

My dataset has a dataframe form with rows the texts and coloums the labels, original text, preprocessed text, etc.

The code that I have is this one

# cleaning texts
import pandas as pd
import re
import nltk
from nltk.corpus import stopwords
from nltk.stem.porter import PorterStemmer
from sklearn.feature_extraction.text import CountVectorizer
corpus = []
for i in range(0, len(df)):
    text = df['Preprocessed_Text'][i]
    text = ''.join(text)
# creating bag of words model
cv = CountVectorizer(max_features = 1500)
X = cv.fit_transform(corpus).toarray()
y = df.iloc[:, 5].values
# splitting the data set into training set and test set
from sklearn.cross_validation import train_test_split
X_train, X_test, y_train, y_test = train_test_split(
           X, y, test_size = 0.25, random_state = 0)
from sklearn import naive_bayes
# fitting naive bayes to the training set
from sklearn.naive_bayes import MultinomialNB
from sklearn.metrics import confusion_matrix
#classifier = GaussianNB()
classifier = MultinomialNB()
classifier.fit(X_train, y_train)
classifier.score(X_test, y_test)

I have the following two questions:

  1. In the pre-processed part the is it correct this one text = ''.join(text)? Or a space (text = ' '.join(text)) is needed?

  2. By this approach we apply first the bag-of-words in the whole dataset and then split it into training and testing. Is there any way to do this in the opposite way? Therefore, first splitting the texts into training and testing and then apply bag of words.

Update: I have already split the texts into training validation and testing in a listing form, thus

# evaluation sets



Then I'm applying the Bag of Words method only in the training and transform the other evaluation sets with the following way

# creating bag of words model
from sklearn.feature_extraction.text import CountVectorizer

cv = CountVectorizer(max_features = 1500)

bow_train = cv.transform(texts_train)
bow_vald = cv.transform(texts_valid)
bow_test = cv.transform(texts_test)

Last step is to fit a classifier and measure its accuracy

# Fitting naive bayes to the training set
from sklearn import naive_bayes
from sklearn.naive_bayes import MultinomialNB

classifier = MultinomialNB()
classifier.fit(bow_train, authors_train)
classifier.score(bow_vald, authors_valid)

2 Answers 2

  1. Assuming that the Preprocessed_Text column contains a regular string, you don't have to do any kind of join since you variable text is a single string.
  2. It's indeed recommended to calculate the bag of words representation only on the training set. It's "cleaner" in the sense that it prevents any possible data leakage, and it's more coherent with respect to applying the model to any fresh test set. However in this case there might be out-of-vocabulary words in the test set, this is normal. You can do this by splitting the "preprocessed text" data first, then calling cv.fit_transform() only on the training instances. Later the test set instances are encoded using cv.transform(), which just maps the new text using the previous representation.
  • $\begingroup$ Thank you so much for your help. It is really valuable. I changes my code and I was wondering if you can check the update on my post, just to make sure that everything is good. $\endgroup$
    – John Smith
    Mar 8, 2022 at 10:34
  • $\begingroup$ @JohnSmith looks good, I don't see anything wrong. Note that accuracy is not a very good evaluation measure if the classes are imbalanced, in this case you could use the more advanced metrics precision, recall, f1-score. $\endgroup$
    – Erwan
    Mar 8, 2022 at 11:38
  • $\begingroup$ yeah that's true, thank you so much! $\endgroup$
    – John Smith
    Mar 8, 2022 at 13:21
  1. You should use space in the join statement. Otherwise the returned text will not be readable as it will show without any space in between the words Example - With ' '.join(text) - It is raining With ''.join(text) - Itisraining
  2. You can do both way.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.