How does tf.random.log_uniform_candidate_sampler work while generating negative samples?

Question

I was trying to build a simple negative sampler for a Word2Vec model using TensorFlow by following the tutorial here.

From what I understand, the tf.random.log_uniform_candidate_sampler() takes as input the correct class to be exluded from sampling, the number of samples to return, and the size of the distribution to sample.

In the tutorial, the true_classes is 1, range_max is 8 and num_ns is 4. Which translates to, "Return 4 samples from the distribution [0, 8), but skip 1, since it's a positive class."

However, in the tutorial, the sampler returns the following tensor: tf.Tensor([2 1 4 3], shape=(4,), dtype=int64).

My question is, why does the sampler return the class 1, even after being told to exclude it, since it doesn't constitute a negative sample?

Furthermore, in the following section of the tutorial:

target_index    : 7
target_word     : sun
context_indices : [1 2 1 4 3]
context_words   : ['the', 'wide', 'the', 'shimmered', 'road']
label           : [1 0 0 0 0]

We can see clearly that the first instance of class 1 (index 0 in context_indices) has been flagged as a positive skip-gram, whereas the second instance of 1 (index 2 in context_indices) has been labelled as a negative skip-gram.

I'd love some clarification or a correction in my understanding of this model or tutorial.

Thanks!

Answer 1

I was trying to go through the same tutorial and I had the same question in mind. I did some research in the codebase and the docs and have the following pointers.

• The tf.random.log_uniform_candidate_sampler function is a sampler that samples classes from an approximately log-uniform or Zipfian distribution. Due to the fact that the words are in a lexicon sorted in decreasing order of frequency, the best way to randomly sample negative words would be to use the function.
• This function does not suppress the positive class. It takes the positive class tensor so that it can return the true_expected_count.

This is the reason why we see an overlap of words in the positive and the negative pair.

Answer 2

This parameter isn't used in this tutorial, is works other return values. you can understand this by read the 122ed line of the source code .

source code