0
$\begingroup$

I would like to predict if an email is spam or not spam based on the information that I have, i.e. date, email address, subject and text.

Three of these parameters are text data, so they would need to encode them. Let’suppose the model considered is SVM (I read this algorithm is very common in text classification), could you please provide me an example of how it works (encoding part)?

What I would need is an explanation of steps required to considered variables that may be important for prediction, for example: subject, text, key words, email address (for instance: @paybal could be a misleading address). It would be great if you could show me how to integrate the variables into the model (numerical and text)

Improve this question
$\endgroup$
1

1 Answer 1

Reset to default
1
$\begingroup$

To use SVM you have to first convert your text features to numeric. You can for a start use the below features:

  1. Date: Day of the Week, Month, Day of the Month all encoded as numbers.
  2. email address: Make a list of popular domains (ex: gmail, hotmail, yahoo) and an unknow (UNK) domain. Encode the email address domain as hot one encoding of this list.
  3. subject : Use sklearn CountVectorizer or TfidfVectorizer to convert subject into a vector. Put a limit of vocabulary size to have a manageable vector size.
  4. text: Do same a subject.
  5. Additional Features: Make a list of possible spam words, and see if they appear in subject/text. Also have a count of how many times they appear.
  6. Additional Features: Is the subject/text all caps ? what is the % of all caps in the subject/text.

You can start by these features and train a SVM.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thank you mujjiga. how should I include, after considered them into the model? For example: let’s say I have columns for month (January,..., december), day (Monday, ..., Sunday), provided (gmail, yahoo,...) and text (dear John, I hope this email finds you well), I should apply countvectorizer to each single column, or the whole dataset after removing numerical columns? I know that for running the model I need the dataset, an x and a y variable (which should be spam(1) not spam (0)) $\endgroup$
    – Math
    Aug 21, 2020 at 11:10
  • 1
    $\begingroup$ Encode the features of date in numbers (say Monday=1.., January=1 so you have two numeric features for date here). Similarly for email id hot one encode to known domain + UNK. CountVectorizer anyway gives a numeric vector. $\endgroup$
    – mujjiga
    Aug 21, 2020 at 11:16
  • $\begingroup$ Thank you so much mujjija. Last question. So once I have my dataset,which shows encoded data day =1 to 7; month = 1 to 12; domains as real numbers; text as numeric vector, should I add the whole dataset in the SVM as df? My difficulties is in understanding how to use it in my SVM code (parameters required) $\endgroup$
    – Math
    Aug 21, 2020 at 11:23

Not the answer you're looking for? Browse other questions tagged or ask your own question.