2
$\begingroup$

I already asked this question but I worded it in such a way that it was a completely different question to the one I want to ask. I have not deleted the old question in case someone finds it useful.

Given two recommendation systems, system A and system B. Users will listen to a number of songs from both systems, and rate each song out of 6. I will then group the ratings based on if they are system A or B, and find the average rating to evaluate which system was better.

I would like to know which is the best system of interlacing the recommendations by A and B, such that bias is minimised.

I have currently implemented 2 methods. The first is that each user only ever listens to the recommendations by one system, so the songs are all AAAAA or BBBB. This leads to the issue that if A is objectively better than B, then if the user rated B a 3, for example, if they had also listened to A, the rating for B might have been lower since it is relatively better. The same applies if the user only listens to A, and the average rating is 3, for example. If they had listened to B, which is worse, it might have received a higher rating since people are not completely objective. Thus, this method introduces a bias against the better system.

The second method is that the users listens to the songs with a 50/50 chance of them being from either system. AABBABABBAA etc. This system eliminates the first system's bias issue, however it introduces another one. If the good system is making good recommendations, and the bad system makes a bad recommendation, the user might allow a better rating for it since they were 'put into a good mood' by the good recommendations. The same applies if the bad system keeps on making bad recommendations, the good system's ratings will be weighted down since the bad system will 'put the user into a bad mood'. Thus, this method also introduces a bias against the better system.

Am I wrong in my reasoning, and if not, is there a better way to interlace the results from both systems while minimising bias?

Thanks

Improve this question
$\endgroup$
3
  • $\begingroup$ Usualy this is done with A/B testing $\endgroup$
    – Nikos M.
    May 4, 2021 at 15:31
  • $\begingroup$ For any user testing the system, should all the responses be from the same system (AAA or BBB) or should they be a mix (ABAB) ? $\endgroup$
    – Michael Pulis
    May 7, 2021 at 9:40
  • $\begingroup$ In A/B testing some users get only A and some users get only B, then you compare A and B from different users $\endgroup$
    – Nikos M.
    May 7, 2021 at 15:39

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.