I am fairly new to machine learning and NLP in general. I am trying to wrap my head around how to do proper text pre-processing (cleaning the text).
I have built a custom text classification model. I have below method that I run on all input text, before serving it to my model. (both in training and testing).
The method will remove stopwords, punctuations and lemmatize the text.
import spacy
from spacy.lang.en.stop_words import STOP_WORDS
import string
def normalize(text, lowercase, remove_stopwords, remove_punctuation):
nlp = spacy.load("en_core_web_sm", disable=['parser', 'tagger', 'ner'])
stops = spacy.lang.en.stop_words.STOP_WORDS
if lowercase:
text = text.lower()
text = nlp(text)
if remove_punctuation:
text = [t for t in text if t.text not in string.punctuation]
lemmatized = list()
for word in text:
lemma = word.lemma_.strip()
if lemma:
if not remove_stopwords or (remove_stopwords and lemma not in stops):
lemmatized.append(lemma)
return " ".join(lemmatized)
Consider below input string:
Input: You're very beautiful!
If I clean that text, using my method:
test_text = "You're very beautiful!"
test_text = normalize(test_text, lowercase=True, remove_stopwords=True, remove_punctuation=True)
It will return: -PRON- beautiful
Is this a "good" way to clean the text? I notice that -PRON-
is kept and therefore also used when training the model (same goes for when testing the model, as I use the same normalize method).
Should the -PRON-
also be removed from the text? And should I complete even more text preprocessing?