I am building a search recommendation system for e-commerce which generates most relevant results given an input query. I have framed it as a classification problem (learning to rank) and using pointwise ranking to compute relevance scores.
Let us say for every input query Q1, we have to generate 30 most relevant results. Suppose, a user initiates a search query and chooses the 3rd result as a relevant result. My question is how should this interaction be used to generate training dataset for the model :
- Q1 - Result 3 : Positive Label
- Q1 - Result 1,2 : Negative Label (User obviously saw result 1,2 and didn't like it. Also mentioned as skip-highs in some papers/blogs).
- Q1 - Random : Negative Label (just add some random results to add random negatives to the dataset).
What about ranks 4-30. It's possible user saw ranks 4-8, but after those ranks user didn't even see the results. Should these lower ranks be used to generate implicit negatives to feed to the model ?
- Random sampling from implicit negatives in lower ranks will ensure that the model is trained on the entire feature distribution that it would have to score during predictions. If we just use the explicit positives/negatives from Step 1/2, model would have been trained only on very relevant results. Sampling from lower ranks will ensure that the model will be trained on entire spectrum of relevant results too.
- We are adding noise to the training dataset, since the user didn't even interact with these results.
So, the main question is should we include the results from rank 4-30, which were probably not even seen by the user as negative examples in the training dataset or not?