I am fairly new to machine learning and NLP in general. I am trying to wrap my head around how to do proper text pre-processing (cleaning the text).

I have built a custom text classification model. I have below method that I run on all input text, before serving it to my model. (both in training and testing).

The method will remove stopwords, punctuations and lemmatize the text.

import spacy
from spacy.lang.en.stop_words import STOP_WORDS
import string

def normalize(text, lowercase, remove_stopwords, remove_punctuation):
    nlp = spacy.load("en_core_web_sm", disable=['parser', 'tagger', 'ner'])
    stops = spacy.lang.en.stop_words.STOP_WORDS

    if lowercase:
        text = text.lower()
    text = nlp(text)
    if remove_punctuation:
        text = [t for t in text if t.text not in string.punctuation]
    lemmatized = list()
    for word in text:
        lemma = word.lemma_.strip()
        if lemma:
            if not remove_stopwords or (remove_stopwords and lemma not in stops):

    return " ".join(lemmatized)

Consider below input string:

Input: You're very beautiful!

If I clean that text, using my method:

test_text = "You're very beautiful!"
test_text = normalize(test_text, lowercase=True, remove_stopwords=True, remove_punctuation=True)

It will return: -PRON- beautiful

Is this a "good" way to clean the text? I notice that -PRON- is kept and therefore also used when training the model (same goes for when testing the model, as I use the same normalize method).

Should the -PRON- also be removed from the text? And should I complete even more text preprocessing?


1 Answer 1


What you are doing seems fine in terms of preprocessing. Removing less informative words like stopwords, punctuation etc. is a very common technique. Here are some of my notes:

  • probably best for speed to load your "nlp" object outside of the function call
  • "-PRON-" must be the lemma for "you're" in this case. So you shouldn't remove it as it is following the correct logic. Though, you should investigate whether your model improves if you change your preprocessing logic
    • if you use SpaCy to create your classification model, I'd recommend experimenting training a model without using any preprocessing. As far as I know, SpaCy handles some of this implicitly through feature engineering
  • $\begingroup$ Thank you for your reply! I'm glad I am on the right path. I have a few counter questions. - Why is it faster to load the nlp() object outside the function call? - How can I prepare two models to figure out what model works the best? (One with pre processed data and one without). Does spaCy offer some insights into the models? $\endgroup$
    – oliverbj
    May 22, 2020 at 13:03
  • 1
    $\begingroup$ It's faster because you only need to load the model once, not at every function call. Plus, if it's inside the function, you need to load it another time if you plan on using it elsewhere. This is just a software engineering tip. To compare models, just create two, train them, and evaluate them on test data. This will tell you how they compare versus each other $\endgroup$ May 22, 2020 at 13:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.