I've written the code below to try word2vec implementation of gensim. I've two questions:

  1. Even though I've removed stop words, the word "the" is listed as one of the most similar words of "friend".
  2. The most similar words of "friend" is not satisfying (at least according to my subjective evaluation). Should I try a larger text (austen-emma.txt file contains 192427 words) or the problem is something else?


import nltk
from nltk.tokenize import sent_tokenize
from nltk.corpus import gutenberg
import gensim 
from gensim.models import Word2Vec 
from gensim.parsing.preprocessing import remove_stopwords
from nltk.tokenize import RegexpTokenizer

text = gutenberg.raw('austen-emma.txt'). 
text = remove_stopwords(text). 
tokenizer = RegexpTokenizer(r'\w+'). 
data = []. 

for i in sent_tokenize(text):     
    temp = []. 
    for j in tokenizer.tokenize(i):   

model = gensim.models.Word2Vec(data, min_count = 1,  
                          size = 32, window = 2) 

model.wv.most_similar(positive='friend', topn=10)

[('mind', 0.9998476505279541),  
 ('present', 0.9998302459716797),  
 ('till', 0.9998292326927185),  
 ('herself', 0.9998183250427246),  
 ('highbury', 0.999806821346283),  
 ('the', 0.9998062252998352),  
 ('place', 0.9998047351837158),  
 ('house', 0.999799907207489),  
 ('her', 0.9997915029525757),  
 ('me', 0.9997879266738892)]  
  • $\begingroup$ Did you check whether your stopwords file has the in it? $\endgroup$ Commented Jul 20, 2020 at 11:37
  • 2
    $\begingroup$ Naive guess, but is stop-word removal case sensitive in gensim? Maybe The is sneaking in which is later getting lower cased? $\endgroup$
    – hssay
    Commented Jul 20, 2020 at 12:39
  • $\begingroup$ @hssay exactly, I should first lower case then call the remove_stopwords function. Any idea about my second question? (fixing stop words issue did not change the results much). $\endgroup$
    – Sanyo Mn
    Commented Jul 20, 2020 at 14:05
  • $\begingroup$ I added it as answer, to make it easy for others to review/weigh-in. $\endgroup$
    – hssay
    Commented Jul 24, 2020 at 11:04

1 Answer 1

  1. Run the lower casing operation before running stop-word removal since it is case sensitive.
  2. Your window size, currently set to 2 is probably too low. Run it with default setting of 5. It will allow more natural similar pairs to be discovered (words frequently found in similar context but with distance more than 2). It would be also helpful to increase the minimum count threshold to weed out low frequency words which can give rise to spurious correlation.

In general, word2vec needs some manual tuning before the results start subjectively making sense (unless your dataset is very large).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.