3
$\begingroup$

I was trying to build a simple negative sampler for a Word2Vec model using TensorFlow by following the tutorial here.

From what I understand, the tf.random.log_uniform_candidate_sampler() takes as input the correct class to be exluded from sampling, the number of samples to return, and the size of the distribution to sample.

In the tutorial, the true_classes is 1, range_max is 8 and num_ns is 4. Which translates to, "Return 4 samples from the distribution [0, 8), but skip 1, since it's a positive class."

However, in the tutorial, the sampler returns the following tensor: tf.Tensor([2 1 4 3], shape=(4,), dtype=int64).

My question is, why does the sampler return the class 1, even after being told to exclude it, since it doesn't constitute a negative sample?

Furthermore, in the following section of the tutorial:

target_index    : 7
target_word     : sun
context_indices : [1 2 1 4 3]
context_words   : ['the', 'wide', 'the', 'shimmered', 'road']
label           : [1 0 0 0 0]

We can see clearly that the first instance of class 1 (index 0 in context_indices) has been flagged as a positive skip-gram, whereas the second instance of 1 (index 2 in context_indices) has been labelled as a negative skip-gram.

I'd love some clarification or a correction in my understanding of this model or tutorial.

Thanks!

$\endgroup$
1
$\begingroup$

I was trying to go through the same tutorial and I had the same question in mind. I did some research in the codebase and the docs and have the following pointers.

  • The tf.random.log_uniform_candidate_sampler function is a sampler that samples classes from an approximately log-uniform or Zipfian distribution. Due to the fact that the words are in a lexicon sorted in decreasing order of frequency, the best way to randomly sample negative words would be to use the function.
  • This function does not suppress the positive class. It takes the positive class tensor so that it can return the true_expected_count.

This is the reason why we see an overlap of words in the positive and the negative pair.

$\endgroup$
0
$\begingroup$

This parameter isn't used in this tutorial, is works other return values. you can understand this by read the 122ed line of the source code .

source code

$\endgroup$
2
  • $\begingroup$ One line and link only answers tend to get flagged for deletion. It is helpful to provide a bit more detail when providing answers if possible. $\endgroup$
    – Ethan
    Dec 7 '20 at 16:14
  • $\begingroup$ @Ethan thank you for you advice, but english is not my mother tongue and I'm not really good at it. I just try my best to provide helps. $\endgroup$
    – dec dec
    Dec 9 '20 at 2:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.