I'm trying to train a recommender model using In-batch Random Negative Sampling as described in the following paper: https://arxiv.org/pdf/2102.06156.pdf. I'm having a bit of difficulty wrapping my mind around the authors' method. In particular, this sentence "We chose to sample 3000 negatives for each positive item, and use 600 as our batch size". Does this mean that in each batch, there are 600 randomly selected positive examples, and 3000 negatives for each positive? Or rather there are 600 pairs of negative and positive, where the positive is a single example randomly selected from 3000 possible negatives for that particular positive example?
3 Answers
$\begingroup$
$\endgroup$
1
From your words, I guess the authors mean that each sample is formed by 3000 negatives and 1 positive, and so each batch is formed by 600(3000+1) examples.
Indeed, the authors write that positive dataset is formed by 10 millions of items, while the negative dataset is formed by around a billion of items, so they form a sample with 3000 negatives and 1 positive, and to maximize the GPU capacity they raise the batch size to 600, so they train a lot of examples in a batch.
New contributor
Iya Lee is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
-
$\begingroup$ I'm not sure then how this corresponds to "in-batch sampling" $\endgroup$– mmmmoMar 15 at 5:04
$\begingroup$
$\endgroup$
The authors mean that in each batch, there are 600 pairs, where each pair consists of one positive example (selected randomly from the set of positive examples) and 3000 negative examples (selected randomly from the set of negative examples for that particular positive example).
In other words, they used a total of 600 pairs (300 positive and 3000 negative examples per positive item) for training the neural network model.
$\begingroup$
$\endgroup$
In the paper you mentioned, the authors are using In-batch Random Negative Sampling (IRNS) for training a recommendation model. IRNS is a technique for training recommender models using negative sampling to improve model performance. Each training instance in the batch consists of a positive item and a negative item. The positive item is the item that the model should recommend, and the negative item is an item that the model should not recommend. In IRNS, the negative item is randomly selected from a set of candidate negative items.
To answer your question,
We chose to sample 3000 negatives for each positive item, and use 600 as our batch size
The authors' statement means that for each positive item in a given batch, they randomly sample 3000 negative items to use for training. So, for a given batch size of 600, there will be 600 positive items, each paired with 3000 negative items, resulting in a total of 1.8 million training instances (600 x 3000) in the batch.
To clarify further, each batch consists of 600 training examples, where 300 of these are positive examples (i.e., items that the user has interacted with, such as viewed or purchased) and 300_10 = 3000 are negative examples (i.e., items that the user has not interacted with). For each positive item, the authors randomly select 10 negative items (from the pool of 3000 negatives) to pair with the positive item, resulting in a total of 300_10 = 3000 negative examples in the batch.
So, in each batch, there are 600 training examples, consisting of 300 positive examples and 3000 negative examples (10 negatives for each positive). The 300 positives and 3000 negatives are randomly sampled for each batch, so the specific examples included in each batch will be different each time the model is trained.
