Let's say I have a model that makes a prediction per individual. An example data set is below. Normally, evaluation metrics (for example within the XGBoost algorthim), are used at the individual observation. However, I don't care whether individual predictions are accurate, I just want predictions in aggregate to be accurate.
In other words, using the sample data set below, and aggregating predictions by Gender, I want to minimize the RMSE between (1000+3000) and (900+3100) for males, and (2000+4000) and (1900+4100) for females; this is as opposed to minimizing the RMSE in the normal sense per each observation.
Is there a phrase for this type of evaluation metric? Is there a way to imlement this into popular algorithms like XGBoost?
Note: There is no reasonable way to aggregate the data prior to training, and ultimately I do need predictions per individual.