I am unable to find details for pretrained word2vec models. If someone is about to use a pre-trained model should it be clear what kind of pre-processing was done before the model was trained? Such as if lower cased was applied or removal of specific stopwords?
- Glove: Is not referring to any kind of that information except of recommending stanford tokenizer (splitting words).
- Google pre trained model: Doesn't give any info for pre-processing of the pretrained model. Refers in a sentence to use a script for the wikipedia training data, but do not mention how the google news were preprocessed before trained.
And in other cases of pretrained models I cant find this kind of information. Someone could assume that there is a specific way of pre-processing for word2vec but I remember that the instructor of stanford NLP course, Richard Socher, that some people are removing stopwords and some others do not.
I have also noticed that people are using different pre-processing methodology and same pre-trained model for the kaggle competition Bag of Words Meets Bags of Popcorn which doesnt make sense for me.
Any one can give some insights?
vec('apple') == vec('Apple')
. $\endgroup$ – Adam Bittlingmayer Dec 4 '16 at 20:50