# When to model a problem by using the Bayes' theorem?

I have a labeled training dataset where each observation has a sentence either in English or in French as its predictors and its label (target value) is whether this sentence is English or French. The test set includes again some sentences either in English or in French but without labels.

A friend of mine suggested that we should model this problem by using the Bayes' theorem since we have have some prior values (labeled observations in training set). I agree that this can work too but I cannot really understand his argument "we should model this problem by using the Bayes' theorem since we have have some prior values".

This is because in my mind every labeled observation can be considered as a prior value and every prior value can be considered as a labeled observation so you can also apply any machine learning classification algorithm e.g. decision trees) in these cases.

Is this right in general or at least for this specific problem?

Why Bayes' theorem modeling comes up as the best solution for the problem which I described above?

• In my experience with Bayes Theorem and other classification algorithms, in couple of cases Naives Bayes out performed rest algorithms. One of the example which might be helpful to you as well is, when I was doing sentiment mining on twitter tweets then the Naives Bayes gave me 86% accuracy literally in most of the cases, on the other hand SVM, Logistic and many more failed to reach 70%. The reason behind that is it works on probability to predict the class of unknown data set. Jul 30, 2018 at 5:42
• Bayes is good not only when features are independent, but also when dependencies of features from each other are similar between features. You can go through the answer given by jb.. Let me know if you need any additional information. Jul 30, 2018 at 5:43
• Thank you for your comments @Toros91. To be honest, I had also something more to ask but perhaps I did not state the whole thing very clearly. Is there any difference between modeling a problem according to the Bayes' theorem and applying the Naive Bayes algorithm? (Is a possible difference that the unconditional probabilities in the former case can be from the whole population whereas in the latter case these will be only from the dataset?) Jul 31, 2018 at 0:18
• hey, you can go thorough this link to understand more about their differences. For the part which you were asking I'm not 100% sure but what you said is right I think. Jul 31, 2018 at 1:50
• Thanks for the link. It was pretty interesting. In other words my question is the following. if I directly apply the Naive Bayes algorithm at a given dataset then the unconditional probabilities (from the Bayes' formula) are automotically taken from this dataset. However, if I know the unconditional probabilties of the whole population (e.g. I know that the probability to choose a certain letter from the English alphabet is 1/26) why not to model the problem according to the Bayes' theorem so that I can fill the (unconditional) probablity values by myself? Jul 31, 2018 at 12:38

You can model your problem with Bayes' theorem. In particular, naive Bayes classifer can be used for binary classification of text data.

Priors in a naive Bayes classifer refers to the base rate for the different classes (i.e., Are there more English or French sentences?)

Naive Bayes may or may not be the best solution. Typically, the best solution is chosen empirically by using predictive performance on a hold-out dataset.

Bayesian classifiers are very good at using words to tokens to classify. use MultinomialNB like any other classifier to predict outcomes. I found for my case that LogisticRegression outperform MultinomialNB, but your dataset may produce different results

paragraph="Herbert Simon research and concepts increased computer scientist understanding of reasoning and increased the computer's ability too solve problems and proof theorems . Herbert Simon , Al Newell , Clifford Shaw proposals were radical and affect computer scientist today . In Simon’s book , “Models of my life” , Simon demonstrated the Logical Theorem algorithm could prove certain mathematical theorems . Simon said , “This was the task to get a system to discover proof for a theorem , not simply to test the proof . We picked logic just because I happened to have Principia Mathematica sitting on my shelf and I was using it to see what was involved in finding a proof of anything . ” Alfred North Whitehead and Bertrand Russell book Principia Mathematica contained theorems considered to form the foundation of mathematical logic . Simeon evolved Logic theorem into General problem solver . GPS is currently used in robotics and gives the robot amazing problem solving capabilities . Many mathematicians considered some of LTs proofs superior to those previously published"

sentences = nltk.sent_tokenize(paragraph)

words=[]
for sentence in sentences:
word_list=nltk.word_tokenize(sentence)

#print(word_list)
for i in range(0, len(word_list)-1):
words.append(word_list[i])

print(words)

def return_weights(vocab, original_vocab, vector, vector_index):
zipped = dict(zip(vector[vector_index].indices, vector[vector_index].data))

# Let's transform that zipped dict into a series
zipped_series = pd.Series({vocab[i]:zipped[i] for i in vector[vector_index].indices})

# Let's sort the series to pull out the top n weighted words
zipped_index = zipped_series.sort_values(ascending=False).index

return [original_vocab[i] for i in zipped_index]
NUMERIC_COLUMNS=[]
LABELS=[]
def combine_text_columns(data_frame, to_drop=NUMERIC_COLUMNS + LABELS):
""" converts all text in each row of data_frame to single vector """

# Drop non-text columns that are in the df
to_drop = set(to_drop) & set(data_frame.columns.tolist())
text_data =data_frame.drop(to_drop,axis=1)

# Replace nans with blanks
text_data.fillna("",inplace=True)

# Join all text items in a row that have a space in between
return text_data.apply(lambda x: " ".join(x), axis=1)

get_text_data=FunctionTransformer(combine_text_columns,validate=False)

pipeline = Pipeline([
('vect', CountVectorizer(stop_words='english',lowercase=True)),
("tfidf1", TfidfTransformer()),
##('vectorizer',TfidfVectorizer(stop_words='english')),
##('chi', SelectKBest()),

('scale', MaxAbsScaler()),
#('clf', LogisticRegression(C=1e5)),
('clf', MultinomialNB())
#('clf', SGDClassifier(loss='hinge', penalty='l2',alpha=1e-3, random_state=42, max_iter=5, tol=None)),
])

sentences = nltk.sent_tokenize(paragraph)

tfidf_vec = TfidfVectorizer(stop_words='english')
text_tfidf = tfidf_vec.fit_transform(sentences)

shape=text_tfidf.get_shape()
vocab= {v:k for k,v in tfidf_vec.vocabulary_.items()}

df=pd.DataFrame(columns=['Index','Text','Tfidf','Target'])
for index in np.arange(shape[0]):
weights=return_weights(vocab,tfidf_vec.vocabulary_,text_tfidf,index)
target=vocab.get(np.max(weights))
index=len(df)
#df.loc[index]=[text_tfidf[index].toarray(),target]
df.loc[index]=[index,sentences[index],text_tfidf[index].toarray(),target]

df.set_index('Index')

#X=df[['Index','Text']].values
#y=df['Target'].values.astype(str)

encoder = LabelEncoder()
df['Target']=encoder.fit_transform(df['Target'])

train,test=train_test_split(df,test_size=.6,random_state=42, shuffle=True)
pipeline.fit(train['Text'],train['Target'])

predictions=pipeline.predict(test['Text'])
print(test['Target'],predictions)

score = f1_score(test['Target'],predictions,pos_label='positive',average='micro')
print("score of Naive Bayes algo is :" , score)