6
$\begingroup$

I am unable to find details for pretrained word2vec models. If someone is about to use a pre-trained model should it be clear what kind of pre-processing was done before the model was trained? Such as if lower cased was applied or removal of specific stopwords?

  • Glove: Is not referring to any kind of that information except of recommending stanford tokenizer (splitting words).
  • Google pre trained model: Doesn't give any info for pre-processing of the pretrained model. Refers in a sentence to use a script for the wikipedia training data, but do not mention how the google news were preprocessed before trained.

And in other cases of pretrained models I cant find this kind of information. Someone could assume that there is a specific way of pre-processing for word2vec but I remember that the instructor of stanford NLP course, Richard Socher, that some people are removing stopwords and some others do not.

I have also noticed that people are using different pre-processing methodology and same pre-trained model for the kaggle competition Bag of Words Meets Bags of Popcorn which doesnt make sense for me.

Any one can give some insights?

Improve this question
$\endgroup$
  • 2
    $\begingroup$ Right, people are not documenting this well. The reason is that these are application-specific optimisations, so you are expected to train your own models anyway. One other library that contains pre-trained word2vec models is spaCy.io. It is well-documented and the developers are responsive on GitHub. $\endgroup$ – Adam Bittlingmayer Dec 4 '16 at 20:50
  • 1
    $\begingroup$ I should also point out that if you are very curious about a specific behaviour, you should just test it. For example, if vec('apple') == vec('Apple'). $\endgroup$ – Adam Bittlingmayer Dec 4 '16 at 20:50
  • $\begingroup$ Here is blog of sebastian ruder , very preciously explain word2vec preprocessing. $\endgroup$ – Abhishek Verma Jun 7 '17 at 3:33
  • $\begingroup$ And Also check stanford Pre trained word embedding. $\endgroup$ – Abhishek Verma Jun 7 '17 at 3:45
1
$\begingroup$

I think in a lot of cases, when people are using pre-trained models, they force their words to lower and singular case and then test to see if the lowercase and singular (not plural) version were trained. Sometimes, you have to check yourself like A.M. Bittlingmayer said.

To provide some insight into why sometimes people remove stopwords versus not --- I think it has to do with whether they want to value infrequent words more or less. Because you don't want to value too highly very frequent words, the algorithm 'punishes' extremely frequent words. So, one could argue that because this already happens, you don't need to remove stopwords since they will be 'punished' anyways. So, not everyone removes stopwords.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.