# spaCy - Text Preprocessing - Keeping "Pronouns" in text

I am fairly new to machine learning and NLP in general. I am trying to wrap my head around how to do proper text pre-processing (cleaning the text).

I have built a custom text classification model. I have below method that I run on all input text, before serving it to my model. (both in training and testing).

The method will remove stopwords, punctuations and lemmatize the text.

import spacy
from spacy.lang.en.stop_words import STOP_WORDS
import string

def normalize(text, lowercase, remove_stopwords, remove_punctuation):
nlp = spacy.load("en_core_web_sm", disable=['parser', 'tagger', 'ner'])
stops = spacy.lang.en.stop_words.STOP_WORDS

if lowercase:
text = text.lower()
text = nlp(text)
if remove_punctuation:
text = [t for t in text if t.text not in string.punctuation]
lemmatized = list()
for word in text:
lemma = word.lemma_.strip()
if lemma:
if not remove_stopwords or (remove_stopwords and lemma not in stops):
lemmatized.append(lemma)

return " ".join(lemmatized)


Consider below input string:

Input: You're very beautiful!

If I clean that text, using my method:

test_text = "You're very beautiful!"
test_text = normalize(test_text, lowercase=True, remove_stopwords=True, remove_punctuation=True)


It will return: -PRON- beautiful

Is this a "good" way to clean the text? I notice that -PRON- is kept and therefore also used when training the model (same goes for when testing the model, as I use the same normalize method).

Should the -PRON- also be removed from the text? And should I complete even more text preprocessing?

• Thank you for your reply! I'm glad I am on the right path. I have a few counter questions. - Why is it faster to load the nlp() object outside the function call? - How can I prepare two models to figure out what model works the best? (One with pre processed data and one without). Does spaCy offer some insights into the models? May 22 '20 at 13:03