In Binary Classification, AUC is a common metric. However, Group-AUC performs better in some scenario, such as we use AUC grouped by user in recommendation systems.

In the below examples, I show that AUC grouped by user may be a better metric over naive AUC. However, I don't know how to choose the right segmentation for grouped AUC, like user, item or any other dimensions.

Our model can predict quite well for each user, which result in a AUC grouped by user = 1. But if we mix all samples together, the AUC is not 1, since some negative samples have high prediction over positive samples. Example

My question

How can we set "group" when using Group AUC metric?

In different industries or tasks, we may choose different metrics. Is there any methodology behind this?



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.