1
$\begingroup$

Will word2vec fail if sentences contain only similar words, or in other words, if the window size is equal to the sentence size? I suppose this question boils down to whether word to vec considers words from other sentences as negative samples, or only words from the same sentence but outside of the window

$\endgroup$

3 Answers 3

1
$\begingroup$

After some testing, it seems that having window size equal to the sentence length does not cause any issues for the model. It must be using other sentences as negative examples.

$\endgroup$
1
  • 1
    $\begingroup$ Vanilla w2v does NOT produce any negative examples for you. Everything it sees are positive & infers “neural” from the rest. $\endgroup$
    – eliangius
    Feb 18, 2022 at 2:17
1
$\begingroup$

Negative sampling aims at maximizing the similarity of the words in the same context and minimizing similarity if they occur in different sentences. However, instead of doing the minimization for all the words in the dictionary except for the context words, it randomly selects a handful of words depending on the training size and uses them to optimize the objective.

If you make context very large you need a lot of training data and training time to train the model to get to good results. So i think negative samples consider word which are out of context.

$\endgroup$
1
$\begingroup$

You are mechanically fine using sentence size as your window size. All context/target word combinations will be treated as positive cases & things will seem to work.

Logically speaking you also want to make some fraction of negative context/target word pairs to yield better embeddings. You can use the popular negative sampling technique. Roughly, it probabilistically picks word pairs from your vocab that don’t exist in your positive set. Like choosing grammatically impossible combinations: (“summer”, “actively”)

Also, since sentence word counts in natural text varies, using this as a dynamic window size might learn uneven word representations. Not sure if this is good or bad but maybe you could report back what you find?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.