I am working on a small exercise for determining if an email is spam or not. My dataset is the following:
Email Spam 0 Hi, I am Andrew and I want too buy VIAGRA 1 1 Dear subscriber, your account will be closed 1 2 Please click below to verify and access email restore 1 3 Hi Anne, I miss you so much! Can’t wait to see you 0 4 Dear Professor Johnson, I was unable to attend class today 0 5 I am pleased to inform you that you have won our grand prize. 1 6 I can’t help you with that cuz it’s too hard. 0 7 I’m sorry to tell you but im sick and will not be able to come to class. 0 8 Can I see an example before all are shipped or will that cost extra? 0 9 I appreciate your assistance and look forward to hearing back from you. 0
where 1 means spam, 0 not spam. What I have tried is the following:
#Tokenization def fun(t): # Removing Punctuations remove_punc = [c for c in text if c not in string.punctuation] remove_punc = ''.join(remove_punc) # Removing StopWords cleaned = [w for w in remove_punc.split() if w.lower() not in stopwords.words('english')] return cleaned
So I applied the function: df['Email'].apply(fun). Then I converted the text into a matrix as follows:
from sklearn.feature_extraction.text import CountVectorizer mex = CountVectorizer(analyzer= fun).fit_transform(df['Email'])
and split the dataset into train and test:
X_train, X_test, y_train, y_test = train_test_split(mex, df['Email'], test_size = 0.25, random_state = 0)
I applied a classifier (I would apply the logistic regression to determine if an email is spam or not, but I have currently only used Naive Bayes:
from sklearn.naive_bayes import MultinomialNB classifier = MultinomialNB() classifier.fit(X_train, y_train)
Finally I applied the classifier first to the train set, then to the test set:
from sklearn.metrics import classification_report,confusion_matrix, accuracy_score from sklearn.metrics import classification_report,confusion_matrix, accuracy_score pred = classifier.predict(X_test) print(classification_report(y_test ,pred )) print('Confusion Matrix: \n', confusion_matrix(y_test,pred)) print() print('Accuracy: ', accuracy_score(y_test,pred))
The code works but I would like to know how to visually see with an example of new email if this has label 1 or 0. For example: if I have a new email
'Hi, my name is Christopher and I like VIAGRA', how could I determine the label/class?
I feel I am missing something or probably I am following a wrong way to demonstrate this.
My question is the following:
Given this new email:
Hi, my name is Christopher and I like VIAGRA, how can I see if this is spam or not? I have thought about classification but probably my approach is wrong. I would like to have something like:
Email Spam ... Hi, my name is Christopher and I like VIAGRA 1
as this is very similar to the email
'Hi, I am Andrew and I want too buy VIAGRA' (if included in the train set or correctly predicted in the test set).
Maybe what I want to do requires only
tf-idf algorithm or a different approach.
Any advice will be appreciated.