1
$\begingroup$

Does the objective function for model fitting and the evaluation metric for model validation need to be identical throughout the hyperparameter search process?

For example, can a XGBoost model be fitted with the Mean Squares Error (MSE) as the objective function (setting the 'objective' argument to reg:squarederror: regression with squared loss), while the cross validation process is evaluated based on a significantly different metric such as the gamma-deviance (residual deviance for gamma regression)? Or should the evaluation metric match the the objective function as much as possible, hence the root mean square error need to be selected as the evaluation metric?

$\endgroup$
1
  • 1
    $\begingroup$ This is quite common. Despite their shortcomings, threshold-based performance metrics like accuracy are popular to use during cross-validation, yet the loss function is likely to be cross-entropy. $\endgroup$ – Dave Sep 29 '20 at 19:49
1
$\begingroup$

The evaluation metric for model validation has to be the same throughout the the hyperparameter search process in order fairly compare different models.

The objective function for model fitting can be different throughout the hyperparameter search process. During the hyperparameter search process, you can compare different algorithms and each of those algorithms can have different objective functions.

The objective function and evaluation metric should be thought of as complete separate concepts. There can only one objective function per algorithm. However, there can be many evaluation metrics. Objective functions are chosen for computers so they can efficiently and effectively fit the training data. Evaluation metrics are chosen for human stakeholders so they can better understand the impact of the model.

The confusion between the two concepts often comes from just measuring a machine learning system on minimizing error. If the measurement of a machine learning system broadens to include other requirements, then the difference between objective function and evaluation metric is more clear. The objective function is typically setup only to minimize error over the training dataset and work well with the chosen optimization technique. Evaluation metrics can also assess model error but can also include prediction speed, model size, model fairness, and many other requirements.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.