0
$\begingroup$

I have two recommendation systems for musical preference that make a list of predictions for a particular user based on the songs they have saved in their library. The user then rates how good each recommendation was out of 6. I will be evaluating the performance of the recommendation systems based on the average rating given to songs recommended by system A and system B.

Let us use A to denote a song recommended by system A and B to denote a song recommended by system B. For a particular user, should the recommendations be (AAAAAA or BBBBBB) or should they be (ABABABAB)? I implemented the first for now being (AAAAAA or BBBBBB). Thus, in the current system, each respondent will be randomly assigned A or B and only get the recommendations from that system. Is this the right approach or does only recommending 1 system to each respondent bias them against what the other system could have recommended?

Let us assume that B is far better than A. If we only recommend the same system to each user, and a user listens to songs which are all system A, they would never had heard system B, and the ratings of A would probably have been different (lower) if they had listened to the better system too. Is the ABABAB approach the best one? Which is the best method to evaluate the performance of each system while reducing bias?

Thanks.

$\endgroup$
0
$\begingroup$

If user rates how good each recommendation then ABABAB approach is the best one. There could be case when system A becomes better than B overtime, then having 2 system does not make sense.

$\endgroup$
1
  • $\begingroup$ Hi. I have revised my question to make it more clear. Which system, (be it AAAA&BBBBB or ABABA, or maybe something I haven't thought of) is better to evaluate the performance of both systems ? $\endgroup$ May 3 at 16:15
0
$\begingroup$

This seems like a good use case for a Bayesian Bandit using Thompson Sampling.

This will allow you to start with a 50-50 recommendation - something like:

ABABABAB

but eventually end up with AAAAABAAAAAB (mostly As) or BBBBBABBBBB (mostly Bs) based on a user liking one of them more than the other.

You might also end up with ABABAABABABAB (blended) if the users like both.

The approach assumes that you have access to feedback or ratings from your users in real time and have the ability to act on it

Read it up - it is pretty simple to implement.

$\endgroup$
1
  • $\begingroup$ Hi. Sorry if my question was not clear. I don't want to give the users the best possible recommendation. Each user will rate the song they listen to. The goal is to find out how effective each system is by getting the average rating for each system, and I was wondering what the best distribution of A&B such that I reduce bias and get the rating for each system. $\endgroup$ May 3 at 17:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.