Machine Learning - Input Prepocessing - NLP email classification model

Question

So I created a model which classifies emails into different categories, just like a spam filter. I deployed the model as a webservice, no problem with that but I can’t get my head around how I would use it to predicht the output category of a new email. How do I preprocess the new email (subject and message body) to match the input format of the model/webservice ? The model I trained has about 1000 features, corresponding to the 1000 most frequent words in the training dataset. Do I vectorize the new email ? Do I just search for the features/words in the new email ? There is something obvious I’m missing, I think.

I used python, sklearn and pandas/numpy to preprocess and train the model

Answer 1

So, from what I have gathered, you are asking how to preprocess the new (I suppose unobserved) emails, which do not appear in the training set. In that case, you should convert your email text into 1000-dimensional vector, where each value corresponds to a particular feature value.

I am going to go with the basis that you are simply counting the number of times any of the most frequent 1000 words occur in a new email (let's call it $x^{(1)}_{test}$).

To convert the email into a vector form, here is one way of doing it:

import pandas as pd
import numpy as np

words = ["blah", "tea", "tetra", "pak"]

def vectorise_email(words, e_mail):

    """Vectorise email to a vector of word counts based on a list of words.

    :param words: (List of Strings) List of frequent words in training set
    :param e_mail: (String) E-mail string

    :return word_counts: (Numpy Array) containing counts of words based on words list.
    """

    e_mail = pd.DataFrame(e_mail.split())

    e_mail = e_mail[e_mail[0].isin(words)][0].value_counts()
    word_counts = np.zeros(len(words))

    for w_idx, word in enumerate(words):
        word_counts[w_idx] = e_mail.at[word]

    return word_counts

print(vectorise_email(words, "this is a blah tetra pak tea tea blah blahh"))

Here we firstly tokenise sentences into words (I used the standard string split method, but you could use nltk's tokenise methods [https://www.nltk.org/api/nltk.tokenize.html]). Then we convert this list into a pandas DataFrame to use the value_counts method (Ref: https://stackoverflow.com/questions/22391433/count-the-frequency-that-a-value-occurs-in-a-dataframe-column) to get the word counts for those words which appear in the word list. We then complete the vectorisation process by mapping these counts into a Numpy Array where each element in the array corresponds to a particular word count in the input e-mail.

Hope that helps

Answer 2

For any data science project when put into production. One has to follow the same preprocessing techniques used before training in production & testing. There is no difference in techniques while performing prediction in testing/production.

Let's assume you have used this function in preprocessing same can be used in production also as it is just before sending data to the model.

def preprocessing(text):
    text = text.split()
    text = remove_stop_words(text)
    return text