Recommend System AB test metric events

Question

I build personal recomendation system for choosing games. In website on main page on special place there is collection of personal games recomendation. And after AB test(between 2 recommend system) I don't understand, what events I should collect. Only events after click on recomend icon or all events(recommend events plus events without choosing recommend game-user can choose game on other places such as finder)?

For example, one of the metric is sum payment per user per game. Should I collect payments only on recommend games, which was clicked, or all payments? As I understand, in theory, recomend system can give negative effect for site(user see advises and not to click for them). And If I collect only recommend payments, I can skip watching negative effect. On other hand, I build recommend system, and want to see metrics only for players, who click on advice icon.

What way is more correct?

Answer 1

This all comes down to your objectives that come to mind which you could use:

1. We care about the entire website. The new recommender system will boost total payments either directly or indirectly. We do not want the recommender to hurt total payments.
2. We care only about the recommender, perhaps we believe this is the future of our business and we want to eventually replace the entire website. It's critical the recommender directly drive payments.

Either way, I would recommend tracking recommender-associated payments and non-associated payments. It's important to understand direct and ecosystem effects. It's possible the recommender is exceptionally successful at driving clicks and finding users great games, but also finding them games which do not monetize successfully. It's also possible the recommender is so awful that it scares people away from the website. You will want to measure and track both direct and indirect effects, but how you make your decision will depend on the objective you are trying to achieve.