I am building a learning spam/ham email classifier as an assignment. It's not supposed to be a good general classifier, but one that can learn on a small set of labeled emails of a user (approx. 650 - 700 messages) and classify the rest, assuming the distributions of both spam/ham and individual features remain roughly similar.
I started by implementing a simple Naive Bayes with log probabilities based on this: https://www.cs.rhodes.edu/~kirlinp/courses/ai/f18/projects/proj3/naive-bayes-log-probs.pdf
That is my baseline which I am trying to improve. I read multiple papers and articles regarding the issue, I tried implementing a more complex Multinomial NB model but that performed significantly worse.
Currently, the classifier's data only comes from the message body and subject (if present). I also treat messages as sets of words and explicitly remove some words from the vocabulary such as "the", "was" etc.
Given a lot of the messages contain HTML code with links, color codes, URLs and email addresses, which is information I currently don't use and I would like to get information from those as well. However, it's proving extremely challenging to do that in some numerically stable way.
For example, in email addresses and URLs I am looking for extremely obvious keyphrases such as "click", "unsubscribe", "ad", "cash" etc.
I tried adding and subtracting constants to the log probability score, multiplying by some factors such as 0.999/1.001, but nothing seems numerically stable and small changes in factors/constants result in big changes in accuracy.
The quality score by which the assignment is graded assumes a 10x weight for False Positives. My current filter usually produces 0-5 false positives on a dataset of 650 messages, but usually over 30 false negatives. On the most difficult dataset (which is black box for me, don't know what it looks like), I get 94 false negatives.
Would anyone with more experience have some tips how to handle the calculation?