Match between objective function and evaluation metric

Question

Does the objective function for model fitting and the evaluation metric for model validation need to be identical throughout the hyperparameter search process?

For example, can a XGBoost model be fitted with the Mean Squares Error (MSE) as the objective function (setting the 'objective' argument to reg:squarederror: regression with squared loss), while the cross validation process is evaluated based on a significantly different metric such as the gamma-deviance (residual deviance for gamma regression)? Or should the evaluation metric match the the objective function as much as possible, hence the root mean square error need to be selected as the evaluation metric?

Answer 1

The evaluation metric for model validation has to be the same throughout the hyperparameter search process in order fairly compare different models.

The objective function for model fitting can be different throughout the hyperparameter search process. During the hyperparameter search process, you can compare different algorithms and each of those algorithms can have different objective functions.

The objective function and evaluation metric should be thought of as completely separate concepts. There can only be one objective function per algorithm. However, there can be many evaluation metrics. Objective functions are chosen for computers so they can efficiently and effectively fit the training data. Evaluation metrics are chosen for human stakeholders so they can better understand the impact of the model.

The confusion between the two concepts often comes from just measuring a machine learning system to minimize error. If the measurement of a machine learning system broadens to include other requirements, then the difference between the objective function and evaluation metric is more apparent. The objective function is typically set up only to minimize error over the training dataset and work well with the chosen optimization technique. Evaluation metrics can also assess model error but can also include prediction speed, model size, model fairness, and many other requirements.