3
$\begingroup$

Does the objective function for model fitting and the evaluation metric for model validation need to be identical throughout the hyperparameter search process?

For example, can a XGBoost model be fitted with the Mean Squares Error (MSE) as the objective function (setting the 'objective' argument to reg:squarederror: regression with squared loss), while the cross validation process is evaluated based on a significantly different metric such as the gamma-deviance (residual deviance for gamma regression)? Or should the evaluation metric match the the objective function as much as possible, hence the root mean square error need to be selected as the evaluation metric?

$\endgroup$
1
  • 1
    $\begingroup$ This is quite common. Despite their shortcomings, threshold-based performance metrics like accuracy are popular to use during cross-validation, yet the loss function is likely to be cross-entropy. $\endgroup$
    – Dave
    Sep 29, 2020 at 19:49

1 Answer 1

3
$\begingroup$

The evaluation metric for model validation has to be the same throughout the hyperparameter search process in order fairly compare different models.

The objective function for model fitting can be different throughout the hyperparameter search process. During the hyperparameter search process, you can compare different algorithms and each of those algorithms can have different objective functions.

The objective function and evaluation metric should be thought of as completely separate concepts. There can only be one objective function per algorithm. However, there can be many evaluation metrics. Objective functions are chosen for computers so they can efficiently and effectively fit the training data. Evaluation metrics are chosen for human stakeholders so they can better understand the impact of the model.

The confusion between the two concepts often comes from just measuring a machine learning system to minimize error. If the measurement of a machine learning system broadens to include other requirements, then the difference between the objective function and evaluation metric is more apparent. The objective function is typically set up only to minimize error over the training dataset and work well with the chosen optimization technique. Evaluation metrics can also assess model error but can also include prediction speed, model size, model fairness, and many other requirements.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.