2
$\begingroup$

I extracted some other features from my dataset regarding punctuation, capital letters, upper case words. I got these value:

enter image description here

looking at the correlation with my target variable (1=spam, 0=not spam), using .corr() in python. BT stands for binary text, e.g., and BS stands for binary summary, where I assign 1 or 0 based on the presence of a capital letter in the text/summary, or upper case word, or...

Do you think that features like these can be useful in model building? I cannot see very strong correlations, but I would like to determine if an email can be spam or not based also on features like these (number of character/text length; presence of !, upper case words,....).

I have around 1000 emails, but only 50 are spam (maybe too small to extract useful information). However, I had to extract these information, so it is a new dataset, built on my own, so I could not get many more spam emails (and I would like to not use datasets from kaggle, for instance).

What do you think?

Improve this question
$\endgroup$
3
  • $\begingroup$ Pandas .corr() can’t be used for binary vectors because does use pearson, spearman and kendal that all of this methods are not suited for binary vectors ... you need to build a custom function to calculate a meaningful correlation coefficient like Phi coefficient. $\endgroup$
    – n1tk
    Oct 18, 2020 at 20:28
  • 1
    $\begingroup$ ... so corr coefficient matrix you got is not informative. Try a Lasso model and see if this will select this features for the model and check confusion matrix if does capture minority class. Use weight with a balance ration for your dataset and should work. $\endgroup$
    – n1tk
    Oct 18, 2020 at 20:33
  • $\begingroup$ thank you @n1tk. I will review it, trying to apply more meaningful correlations. Thanks a lot! $\endgroup$
    – LdM
    Oct 18, 2020 at 23:48

1 Answer 1

Reset to default
2
$\begingroup$

First about the features i think you could add some such as :

  • the time when the letter is received,
  • number of links in the email,
  • the whole structure (does it follow typical structure for email),
  • number of words that contains numbers in it,
  • what is the whole mood of the email (sales,threats,info,...-for this purpose you can use sentiment analysis),
  • number of attachments ,
  • type of attachments and so on.

After that try with feature selection (you could read more about it here). For the imbalance data you need to resample the data. I would:

  • add copies of spam emails(oversampling)
  • try to generate new spam email (smote)

You can read more here. I hope my answer will give you some clarity.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thank you so much @mariq vlahova (and Erwan for editing her answer) $\endgroup$
    – LdM
    Oct 18, 2020 at 23:49
  • $\begingroup$ Thanks Erwan for editing my comment 🙂 $\endgroup$
    – mariq vlahova
    Oct 19, 2020 at 6:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.