@J_H Sorry I forgot to mention it's not only about spams/hams but about domains & client subnets which are actually sending them.

@J_H I guess I already described implementation.

@J_H Sorry I donot totally get what you mean by buisness domain. But I guess in my case buisness domain is spam email detection, P1 is genuine person sending mail, P2 is fraudster sending mail. About human evaluation I currently have no structured feedback mechanism but rather some more structured rules which can identify part of P1 cases and part of P2 cases with confidence. Does this helps?

@J_H Thanks for guidance, can u look at modified question (Edit part) if should provide more information.