Let's say I have a model that makes a prediction per individual. An example data set is below. Normally, evaluation metrics (for example within the XGBoost algorthim), are used at the individual observation. However, I don't care whether individual predictions are accurate, I just want predictions in aggregate to be accurate.

In other words, using the sample data set below, and aggregating predictions by Gender, I want to minimize the RMSE between (1000+3000) and (900+3100) for males, and (2000+4000) and (1900+4100) for females; this is as opposed to minimizing the RMSE in the normal sense per each observation.

Is there a phrase for this type of evaluation metric? Is there a way to imlement this into popular algorithms like XGBoost?

Note: There is no reasonable way to aggregate the data prior to training, and ultimately I do need predictions per individual.

Sample Data


I don't care whether individual predictions are accurate

If this is true (which I don't think it is), then you could just take the group mean of the target for males and females to minimize the RMSE and make this your prediction for all male and female individuals, respectively. (Basically a model where Gender is the only feature if you insist on using xgboost.) One could argue that this one-feature model might not generalize as well to new data than a model with more features, but if Gender is not in any meaningful way related to the target you can't optimize the RMSE towards Gender anyway.

Based on your "Note" I assume you somehow want both, good predictions for individuals and on the group level. In this case minimizing the normal "individual-level" RMSE is already the best way to go.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.