# Two questions about word2vec and gensim

I've written the code below to try word2vec implementation of gensim. I've two questions:

1. Even though I've removed stop words, the word "the" is listed as one of the most similar words of "friend".
2. The most similar words of "friend" is not satisfying (at least according to my subjective evaluation). Should I try a larger text (austen-emma.txt file contains 192427 words) or the problem is something else?

Thanks.

import nltk
from nltk.tokenize import sent_tokenize
from nltk.corpus import gutenberg
import gensim
from gensim.models import Word2Vec
from gensim.parsing.preprocessing import remove_stopwords
from nltk.tokenize import RegexpTokenizer

text = gutenberg.raw('austen-emma.txt').
text = remove_stopwords(text).
tokenizer = RegexpTokenizer(r'\w+').
data = [].

for i in sent_tokenize(text):
temp = [].
for j in tokenizer.tokenize(i):
temp.append(j.lower()).
data.append(temp).

model = gensim.models.Word2Vec(data, min_count = 1,
size = 32, window = 2)

model.wv.most_similar(positive='friend', topn=10)

[('mind', 0.9998476505279541),
('present', 0.9998302459716797),
('till', 0.9998292326927185),
('herself', 0.9998183250427246),
('highbury', 0.999806821346283),
('the', 0.9998062252998352),
('place', 0.9998047351837158),
('house', 0.999799907207489),
('her', 0.9997915029525757),
('me', 0.9997879266738892)]

• Did you check whether your stopwords file has the in it? Jul 20, 2020 at 11:37
• Naive guess, but is stop-word removal case sensitive in gensim? Maybe The is sneaking in which is later getting lower cased? Jul 20, 2020 at 12:39
• @hssay exactly, I should first lower case then call the remove_stopwords function. Any idea about my second question? (fixing stop words issue did not change the results much). Jul 20, 2020 at 14:05
• I added it as answer, to make it easy for others to review/weigh-in. Jul 24, 2020 at 11:04