How to choose negative examples for recommendation system?

Question

I am building a search recommendation system for e-commerce which generates most relevant results given an input query. I have framed it as a classification problem (learning to rank) and using pointwise ranking to compute relevance scores.

Let us say for every input query Q1, we have to generate 30 most relevant results. Suppose, a user initiates a search query and chooses the 3rd result as a relevant result. My question is how should this interaction be used to generate training dataset for the model :

1. Q1 - Result 3 : Positive Label
2. Q1 - Result 1,2 : Negative Label (User obviously saw result 1,2 and didn't like it. Also mentioned as skip-highs in some papers/blogs).
3. Q1 - Random : Negative Label (just add some random results to add random negatives to the dataset).

What about ranks 4-30. It's possible user saw ranks 4-8, but after those ranks user didn't even see the results. Should these lower ranks be used to generate implicit negatives to feed to the model ?

Pros

• Random sampling from implicit negatives in lower ranks will ensure that the model is trained on the entire feature distribution that it would have to score during predictions. If we just use the explicit positives/negatives from Step 1/2, model would have been trained only on very relevant results. Sampling from lower ranks will ensure that the model will be trained on entire spectrum of relevant results too.

Cons

• We are adding noise to the training dataset, since the user didn't even interact with these results.

So, the main question is should we include the results from rank 4-30, which were probably not even seen by the user as negative examples in the training dataset or not?

Answer 1

Yes, we should include negative samples, more so to correct selection or sampling bias.

The user would have seen only a subset that is selected by another algorithm or rules which decided that the subset is most relevant to the user. Of that, position bias kicks in and the user may not see everything. So the choices within a query or the documents within a query are likely to be very similar to one another (perhaps small differences). This makes it diff for the algorithm to learn the parameters.

Whereas while scoring in production, almost all choices/documents have to be scored for each query. UNLESS the old model is used first to select a subset of hotels and if learning-to-rank model reranks among the subset.

In my experience adding negative samples (that we can safely tell that user would not have picked/clicked the item even if he/she saw the item) helps. But it is difficult to define such base for all use-cases.

We are still experimenting with this.