I have a dataset with three columns "message", "city" and "has_info". Here is a sample of it:
message city has_info
ill be there soon, call me 313-972-0310 New-York 1
use this email john***@gmail.com Boston 1
ok. you can check it Boston 0
................................. ....... ..
i love it Miami 0
has_info column is binary column which defines whether or not some contact information was mentioned in column "message" (1 if there was, 0 if wasn't). I have train and test pandas dataframes like that. And I want to make classifier to predict target value "has_info" in test dataset.
I turned feature "city" into categorical one and created couple of new features as well, like number of words in message for example. I also used bag of words method by finding up to 1000 most frequent tokens in train dataset and sorting them by number of occurences (highest first). So it will create 1000 additional features.
All of it however gave me only AUC value 0.85.
I wanted to know, if there any other (better) method for this particular case? Maybe I should just manually create list of red flag words (phone, mail, number, call, text, etc.) and based on them create dummy variables whether they occurred in message or not? Is there any other nlp solution that can probably give me at least 0.9 AUC?
My train dataset has 900000 rows in it, so its very large.
Thanks in advance