spaCy - Text Preprocessing - Keeping “Pronouns” in text

Question

I am fairly new to machine learning and NLP in general. I am trying to wrap my head around how to do proper text pre-processing (cleaning the text).

I have built a custom text classification model. I have below method that I run on all input text, before serving it to my model. (both in training and testing).

The method will remove stopwords, punctuations and lemmatize the text.

import spacy
from spacy.lang.en.stop_words import STOP_WORDS
import string

def normalize(text, lowercase, remove_stopwords, remove_punctuation):
    nlp = spacy.load("en_core_web_sm", disable=['parser', 'tagger', 'ner'])
    stops = spacy.lang.en.stop_words.STOP_WORDS

    if lowercase:
        text = text.lower()
    text = nlp(text)
    if remove_punctuation:
        text = [t for t in text if t.text not in string.punctuation]
    lemmatized = list()
    for word in text:
        lemma = word.lemma_.strip()
        if lemma:
            if not remove_stopwords or (remove_stopwords and lemma not in stops):
                lemmatized.append(lemma)

    return " ".join(lemmatized)

Consider below input string:

Input: You're very beautiful!

If I clean that text, using my method:

test_text = "You're very beautiful!"
test_text = normalize(test_text, lowercase=True, remove_stopwords=True, remove_punctuation=True)

It will return: -PRON- beautiful

Is this a "good" way to clean the text? I notice that -PRON- is kept and therefore also used when training the model (same goes for when testing the model, as I use the same normalize method).

Should the -PRON- also be removed from the text? And should I complete even more text preprocessing?

Answer 1

What you are doing seems fine in terms of preprocessing. Removing less informative words like stopwords, punctuation etc. is a very common technique. Here are some of my notes:

• probably best for speed to load your "nlp" object outside of the function call
• "-PRON-" must be the lemma for "you're" in this case. So you shouldn't remove it as it is following the correct logic. Though, you should investigate whether your model improves if you change your preprocessing logic
  • if you use SpaCy to create your classification model, I'd recommend experimenting training a model without using any preprocessing. As far as I know, SpaCy handles some of this implicitly through feature engineering