What are the preprocessing steps for text classification after removing stopwords?

Question

I am working on an NLP project where I have text that I need to categorize based on topics (The data is 2 columns, text and topic).

Something that I am stuck on now is the preprocessing part. What are the steps and in what order?

So far, what I have done is remove stopwords.

I have heard of TFIDF, Count Vectorizer, BOW, Tokenization, and Lemmatization/Stemming. However, I am confused about when to use them and in what order. Could someone please explain what can I do after removing stopwords? And do I need to one-hot encode the labels (topics) in order to pass it on to the model?

Thanks in advance.

Answer 1

You are on the right track as far as preprocessing techniques are considered. Generally this order is followed :

1. Lowercase the text if your problem statement allows. In this case you can go ahead and do it.
2. Tokenize the text. This can be sentence level or word level.
3. Lemmatize/Stemming the tokens. This reduces the tokens to their base level.
4. Remove the following :- stops words, punctuation marks, hyperlinks, smileys, email ids etc. Basically anything that is not needed for classification.
5. Vectorize the text using BOW or TFIDF approach.

Cheers!