So I created a model which classifies emails into different categories, just like a spam filter. I deployed the model as a webservice, no problem with that but I can’t get my head around how I would use it to predicht the output category of a new email. How do I preprocess the new email (subject and message body) to match the input format of the model/webservice ? The model I trained has about 1000 features, corresponding to the 1000 most frequent words in the training dataset. Do I vectorize the new email ? Do I just search for the features/words in the new email ? There is something obvious I’m missing, I think.

I used python, sklearn and pandas/numpy to preprocess and train the model

  • $\begingroup$ From a ML perspective, there is no difference between applying a model in production and applying a model to a test set (unseen instances). This implies that any preprocessing should have been included in the design of the model from the start. Maybe that's the issue? $\endgroup$ – Erwan Jun 6 '20 at 18:49

So, from what I have gathered, you are asking how to preprocess the new (I suppose unobserved) emails, which do not appear in the training set. In that case, you should convert your email text into 1000-dimensional vector, where each value corresponds to a particular feature value.

I am going to go with the basis that you are simply counting the number of times any of the most frequent 1000 words occur in a new email (let's call it $x^{(1)}_{test}$).

To convert the email into a vector form, here is one way of doing it:

import pandas as pd
import numpy as np

words = ["blah", "tea", "tetra", "pak"]

def vectorise_email(words, e_mail):

    """Vectorise email to a vector of word counts based on a list of words.

    :param words: (List of Strings) List of frequent words in training set
    :param e_mail: (String) E-mail string

    :return word_counts: (Numpy Array) containing counts of words based on words list.

    e_mail = pd.DataFrame(e_mail.split())

    e_mail = e_mail[e_mail[0].isin(words)][0].value_counts()
    word_counts = np.zeros(len(words))

    for w_idx, word in enumerate(words):
        word_counts[w_idx] = e_mail.at[word]

    return word_counts

print(vectorise_email(words, "this is a blah tetra pak tea tea blah blahh"))

Here we firstly tokenise sentences into words (I used the standard string split method, but you could use nltk's tokenise methods [https://www.nltk.org/api/nltk.tokenize.html]). Then we convert this list into a pandas DataFrame to use the value_counts method (Ref: https://stackoverflow.com/questions/22391433/count-the-frequency-that-a-value-occurs-in-a-dataframe-column) to get the word counts for those words which appear in the word list. We then complete the vectorisation process by mapping these counts into a Numpy Array where each element in the array corresponds to a particular word count in the input e-mail.

Hope that helps

  • $\begingroup$ Thanks for the reply, it really helps, but I feel I'm overlooking something. It's seems like a fairly standard problem, right ? How to use a NLP model in production and classify new items. Seeing emai/text classifiers are failry well documented in literature, there must be some standardized way of using such classifiers as a service and correctly processing incoming emails.But anyway, greatly appreciate your help, brings me alot closer. $\endgroup$ – mmwindel Jun 6 '20 at 13:49

For any data science project when put into production. One has to follow the same preprocessing techniques used before training in production & testing. There is no difference in techniques while performing prediction in testing/production.

Let's assume you have used this function in preprocessing same can be used in production also as it is just before sending data to the model.

def preprocessing(text):
    text = text.split()
    text = remove_stop_words(text)
    return text

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.