How to properly compute spam message score as a combination of fixed features and probability from a naive bayes classifier?

Question

I am building a learning spam/ham email classifier as an assignment. It's not supposed to be a good general classifier, but one that can learn on a small set of labeled emails of a user (approx. 650 - 700 messages) and classify the rest, assuming the distributions of both spam/ham and individual features remain roughly similar.

I started by implementing a simple Naive Bayes with log probabilities based on this: https://www.cs.rhodes.edu/~kirlinp/courses/ai/f18/projects/proj3/naive-bayes-log-probs.pdf

That is my baseline which I am trying to improve. I read multiple papers and articles regarding the issue, I tried implementing a more complex Multinomial NB model but that performed significantly worse.

Currently, the classifier's data only comes from the message body and subject (if present). I also treat messages as sets of words and explicitly remove some words from the vocabulary such as "the", "was" etc.

Given a lot of the messages contain HTML code with links, color codes, URLs and email addresses, which is information I currently don't use and I would like to get information from those as well. However, it's proving extremely challenging to do that in some numerically stable way.

For example, in email addresses and URLs I am looking for extremely obvious keyphrases such as "click", "unsubscribe", "ad", "cash" etc.

I tried adding and subtracting constants to the log probability score, multiplying by some factors such as 0.999/1.001, but nothing seems numerically stable and small changes in factors/constants result in big changes in accuracy.

The quality score by which the assignment is graded assumes a 10x weight for False Positives. My current filter usually produces 0-5 false positives on a dataset of 650 messages, but usually over 30 false negatives. On the most difficult dataset (which is black box for me, don't know what it looks like), I get 94 false negatives.

Would anyone with more experience have some tips how to handle the calculation?

Thanks!

Answer 1

It sounds like you are trying to improve the performance of your spam email classifier by adding additional features to your model. One approach you could consider is to use a feature extractor function to extract useful information from the HTML code and email addresses in the messages. This function could take a message as input and return a feature vector representing the message. You could then use this feature vector as input to your Naive Bayes classifier.

To extract useful features from the HTML code and email addresses, you could consider using regular expressions to match specific patterns. For example, you could use a regular expression to match email addresses that contain certain keyphrases, such as "click", "unsubscribe", "ad", or "cash". You could also use regular expressions to match URLs that contain these keyphrases, or to identify specific HTML tags or attributes that are often used in spam emails.

In addition to using regular expressions, you could also consider using other techniques to extract features from the HTML code and email addresses. For example, you could use natural language processing techniques to identify keyphrases or sentiment in the text of the email, or you could use machine learning techniques to identify patterns or anomalies in the data.

Once you have extracted the features from the messages, you can use these features to update the log probability scores that you are currently using in your Naive Bayes classifier. You can do this by adding or subtracting the log probabilities of the additional features to the log probabilities of the existing features.

It is important to be careful when adding or subtracting log probabilities, as small changes in the log probabilities can lead to large changes in the probability. To avoid this issue, you may want to consider using a smoothing technique, such as Laplace smoothing, to smooth out the log probabilities and make your model more robust to small changes in the data.

Another approach you could try is to use a different classification algorithm that is better suited to handling high-dimensional data and a large number of features. For example, you could try using a linear support vector machine (SVM) or a random forest classifier, which are both known to perform well on high-dimensional data.

If you want to continue using a Naive Bayes classifier, you could try using a Bernoulli Naive Bayes classifier, which is designed for binary data, rather than a Multinomial Naive Bayes classifier, which is designed for count data. This might be more suitable for the type of features you are trying to incorporate.

In terms of improving the performance of your classifier, it might be helpful to try using cross-validation to evaluate the performance of different models and hyperparameter configurations, and to choose the model and configuration that performs best. You could also try using techniques such as feature selection to identify the most important features for your classification task and to reduce the dimensionality of your data.

I hope this helps! Let me know if you have any other questions.