I am studying learning to rank and not sure I understand how the train sample and final label (relevance score) is constructed.

Lets assume we sell furniture online. We have logged customer's query, product customer bought, clicked.

User A searched for "red sofa", clicked on Pa(r=1), Pb(r=5), Pc(r=10) and bought Pc.
User B searched for "red sofa", clicked on Pa(r=1), Pd(r=2), Pe(r=6) and bought Pd.
User C searched for "blue chair", clicked on Pu(r=1), Po(r=2), Ps(r=6) and bought Pu.

(r stands for absolute position from top to bottom of a product). (lets say for clicking we would add 1, for buying we would add 2)

Case 1: Without any personalization, I just want to rank the products only based on textual data from the product and query. How do I construct my training data?

Two users searched for "red sofa", do I treat those separate, e.g per user per query?

user     query         product     relevance  some_features  
A        "red sofa"    Pc             2  
A        "red sofa"    Pa             1   
A        "red sofa"    Pb             1 
A        "red sofa"    other products 0
B        "red sofa"    Pd             2  
B        "red sofa"    Pa             1  
B        "red sofa"    Pe             1  
B        "red sofa"    other products 0
C        "blue chair"  Pu             2  
C        "blue chair"  Po             1  
C        "blue chair"  Ps             1
C        "blue chair"  other products 0  

This looks to me a bit weird because how should an algorithm understand how products Pc or Pd (bought by user A and user B) should be ranked compared to each other?

Or do I treat logs for the same query together, e.g aggregating per query? (Because user A and B clicked both on Pa, it got a relevance of 2=1+1)

query         product     relevance  some_features  
"red sofa"    Pc             2  
"red sofa"    Pa             2   
"red sofa"    Pb             1 
"red sofa"    Pd             2  
"red sofa"    Pe             1  
"red sofa"    other products 0
"blue chair"  Pu             2  
"blue chair"  Po             1  
"blue chair"  Ps             1
"blue chair"  other products 0  

This also looks weird to me because if users click the same products but buy different ones, the relevance of clicked products will be higher.

Question 1: How should I represent a label and a training sample in this case?

Case 2: With personalization. If I want to provide user's features like "clicked before", "put into favorites", I would indeed construct a training record per user, query.

Question 2: Is it correct to do so?

  • $\begingroup$ I believe you need more data or more features. As the data you have shown does not give a complete picture of user behavior. So case 2 becomes more relevant in this suggestion. If you take more data then there are chances of such noise getting removed automatically. $\endgroup$ Jun 9 '19 at 15:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.