2
$\begingroup$

I have text data which is crawled from websites. I am preprocessing data to train Word2Vec model. Should I remove stopwords and do lemmatization? How to preprocess data for Word2Vec?

$\endgroup$
2
  • $\begingroup$ Different libraries require different preprocessing steps. What are you working with? $\endgroup$
    – Leevo
    Feb 13, 2020 at 13:16
  • $\begingroup$ I am using gensim but open to try different libraries too. @Leevo $\endgroup$
    – nehiridil
    Feb 13, 2020 at 13:24

1 Answer 1

3
$\begingroup$

Welcome to the community,

I do not know about other libraries, but gensim has a very good API to create word2vec models. In order to preprocess data, you have to decide first what things you are gonna keep in your vocab and whatnot. for ex:- Punctuations, numbers, alphanumeric words(ex - 42nd) etc.

In my knowledge, the most generic preprocessing pipeline is the following:-

1) Convert to lower 2) Remove punctuations/symbols/numbers (but it is your choice) 3) Normalize the words (lemmatize and stem the words)

Once this is done, now you can tokenize the sentence into uni/bi/tri-grams.

Have a look at this

The generic format to put data in gensim.models.word2vec()'s sentence parameter is : [[tokeneized sentence 1], [tokenized sentence 2].....and so on]

Hope it helps, thanks!!

$\endgroup$
3
  • $\begingroup$ great if this question solves your problem, kindly accept this answer as solution so that the admin can take it off from the open questions domain. Thanks!! $\endgroup$ Feb 14, 2020 at 18:00
  • $\begingroup$ I tried 2 versions of it, one with normalized words another without normalizing. In my case, without normalization model works better. Thanks! $\endgroup$
    – nehiridil
    Feb 16, 2020 at 9:23
  • $\begingroup$ nice, please have look at this to accept this solution as asnwer. stackoverflow.com/help/accepted-answer $\endgroup$ Feb 17, 2020 at 7:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.