Should I prepare my training data for word2vec Skip-gram embedding as unique target-context word pairs discovered throughout the corpus? Or should the repeated occurrences of the same pairs be present multiple times in the training data as well?

I am well aware of the subsampling method, but it is unclear whether I should downsample frequent words (so unique pairs containing them will be sampled less likely into the training) or I should downsample frequent pairs entirely.

My intuition is that if frequent pairs are presented more to the network during training as opposed to infrequent ones, then their embedding will have closer resemblance. However, if I understand right, Skip-gram seems to only differentiate between context and non-context words, and gives no importance to how "close" of a context or non-context they are.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.