I want to use the Naive Bayes model as a baseline in an classification task that I am working. I found this really useful tutorial: https://www.geeksforgeeks.org/applying-multinomial-naive-bayes-to-nlp-problems/ and I want to apply it into my problem.
My dataset has a dataframe form with rows the texts and coloums the labels, original text, preprocessed text, etc.
The code that I have is this one
# cleaning texts import pandas as pd import re import nltk from nltk.corpus import stopwords from nltk.stem.porter import PorterStemmer from sklearn.feature_extraction.text import CountVectorizer nltk.download('stopwords') corpus =  for i in range(0, len(df)): text = df['Preprocessed_Text'][i] text = ''.join(text) corpus.append(text)
# creating bag of words model cv = CountVectorizer(max_features = 1500) X = cv.fit_transform(corpus).toarray() y = df.iloc[:, 5].values
# splitting the data set into training set and test set from sklearn.cross_validation import train_test_split X_train, X_test, y_train, y_test = train_test_split( X, y, test_size = 0.25, random_state = 0)
from sklearn import naive_bayes # fitting naive bayes to the training set from sklearn.naive_bayes import MultinomialNB from sklearn.metrics import confusion_matrix #classifier = GaussianNB() classifier = MultinomialNB() classifier.fit(X_train, y_train) classifier.score(X_test, y_test)
I have the following two questions:
In the pre-processed part the is it correct this one text = ''.join(text)? Or a space (text = ' '.join(text)) is needed?
By this approach we apply first the bag-of-words in the whole dataset and then split it into training and testing. Is there any way to do this in the opposite way? Therefore, first splitting the texts into training and testing and then apply bag of words.
Update: I have already split the texts into training validation and testing in a listing form, thus
# evaluation sets texts_train auhors_train texts_valid authors_valid texts_test authors_test
Then I'm applying the Bag of Words method only in the training and transform the other evaluation sets with the following way
# creating bag of words model from sklearn.feature_extraction.text import CountVectorizer cv = CountVectorizer(max_features = 1500) cv.fit_transform(texts_train).toarray() bow_train = cv.transform(texts_train) bow_vald = cv.transform(texts_valid) bow_test = cv.transform(texts_test)
Last step is to fit a classifier and measure its accuracy
# Fitting naive bayes to the training set from sklearn import naive_bayes from sklearn.naive_bayes import MultinomialNB classifier = MultinomialNB() classifier.fit(bow_train, authors_train) classifier.score(bow_vald, authors_valid)