I am fairly new to machine learning and NLP in general. I am trying to wrap my head around how to do proper text pre-processing (cleaning the text).
I have built a custom text classification model. I have below method that I run on all input text, before serving it to my model. (both in training and testing).
The method will remove stopwords, punctuations and lemmatize the text.
import spacy from spacy.lang.en.stop_words import STOP_WORDS import string def normalize(text, lowercase, remove_stopwords, remove_punctuation): nlp = spacy.load("en_core_web_sm", disable=['parser', 'tagger', 'ner']) stops = spacy.lang.en.stop_words.STOP_WORDS if lowercase: text = text.lower() text = nlp(text) if remove_punctuation: text = [t for t in text if t.text not in string.punctuation] lemmatized = list() for word in text: lemma = word.lemma_.strip() if lemma: if not remove_stopwords or (remove_stopwords and lemma not in stops): lemmatized.append(lemma) return " ".join(lemmatized)
Consider below input string:
You're very beautiful!
If I clean that text, using my method:
test_text = "You're very beautiful!" test_text = normalize(test_text, lowercase=True, remove_stopwords=True, remove_punctuation=True)
It will return:
Is this a "good" way to clean the text? I notice that
-PRON- is kept and therefore also used when training the model (same goes for when testing the model, as I use the same normalize method).
-PRON- also be removed from the text? And should I complete even more text preprocessing?