I have a doubt that we have been discussing for weeks with my colleagues and I wanted your opinion. I have a model for diagnosis of a disease and I want to know if it is fair. I train the model with one cohort and I use another cohort for testing. And I want to evaluate fairness in gender for this disease. My first idea was to downsample the test to have the same number of participants with the disease than healthy. But our discussion takes place because the testing cannot be touched. But this is clear in diagnosis and prediction but not for fairness. Some papers srratify by groups. If I use the full cohort most of the cases are healthy women (negative and sensitive group) in a ratio e.g., of 7.000 vs 100 so the fairness is most showing the bias for the healthy women than what i am more interested women with disease and this is not what i want. I wonder if someone thought about this issue. Fairness is a new issue and i do not see it clearly to treat it as prediction or classification. I think testing should be stratified somehow for not having those imbalance ratios between healthy and non-healthy.
Any help? Thank you