1
$\begingroup$

I am working on an NLP project where I have text that I need to categorize based on topics (The data is 2 columns, text and topic).

Something that I am stuck on now is the preprocessing part. What are the steps and in what order?

So far, what I have done is remove stopwords.

I have heard of TFIDF, Count Vectorizer, BOW, Tokenization, and Lemmatization/Stemming. However, I am confused about when to use them and in what order. Could someone please explain what can I do after removing stopwords? And do I need to one-hot encode the labels (topics) in order to pass it on to the model?

Thanks in advance.

$\endgroup$

1 Answer 1

0
$\begingroup$

You are on the right track as far as preprocessing techniques are considered. Generally this order is followed :

  1. Lowercase the text if your problem statement allows. In this case you can go ahead and do it.
  2. Tokenize the text. This can be sentence level or word level.
  3. Lemmatize/Stemming the tokens. This reduces the tokens to their base level.
  4. Remove the following :- stops words, punctuation marks, hyperlinks, smileys, email ids etc. Basically anything that is not needed for classification.
  5. Vectorize the text using BOW or TFIDF approach.

Cheers!

$\endgroup$
2
  • $\begingroup$ Thank you! The vectorized text in step 5 is what will be passed on to the model, is that correct? Also, could you tell me why do we tokenize before stemming and removing stopwords? $\endgroup$
    – soup
    Jun 16, 2023 at 12:57
  • 1
    $\begingroup$ Yes the vectorized data is paased on to the model as ML models cannot work with categorical/string/text data and need to be vectorized. Stemming/Lemmatization and Stop Word Removal are word level techniques that work on words/tokens and not on paragraphs. Hence the need to tokenize! $\endgroup$
    – spectre
    Jun 16, 2023 at 15:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.