Potential Sign Issues in a Composite Performance Metric for Model Selection

Question

I am analyzing the results of various machine learning models for a regression task, using four metrics: RMSE, MAE, MAPE, and $R^2$. My approach involves two types of analyses:

1. Individual Metric Analysis: Where each metric is considered separately.
2. Composite Performance Metric Analysis: A combined metric to determine the best model among the candidates.

The Composite Performance Metric (CPM) incorporates the standardized mean of the metrics derived from the individual results obtained through cross-validation: $$ \begin{equation} \text{CPM} = M_{norm}(\text{RMSE}) + M_{norm}(\text{MAE}) + M_{norm}(\text{MAPE}) - M_{norm}(R^2) \end{equation} $$

where

$$ \begin{equation} M_{norm}(\text{metric}) = \frac{\text{metric} - \mu_{\text{metric}}}{\sigma_{\text{metric}}} \end{equation} $$ and $\mu_\text{metric}$ is the overall mean of the metric across all models, and $\sigma_\text{metric}$ is the overall standard deviation of the metric across all models.

The CPM prioritizes models with lower errors (RMSE, MAE, MAPE) and higher $R^2$ values. By standardizing each metric, the CPM ensures that all metrics contribute equally, allowing for a fair comparison across different models. It essentially functions as a weighted sum where each metric is equally weighted.

The Issue with Sign Changes

However, I've encountered a potential issue: during the standardization process, the signs of some metrics may change. This could lead to misleading results when calculating the CPM. For instance, if the sign of $R^2$ flips due to standardization, it might inadvertently increase the CPM, penalizing models that should be rewarded for their higher $R^2$ values.

Proposed Solution

To address this issue, I am considering the following approach:

• Error Metrics (RMSE, MAE, MAPE): Since these metrics are non-negative, I propose taking the absolute value of the standardized metrics to ensure they always contribute positively to the CPM.
• $R^2$ Metric: I suggest preserving the original sign of $R^2$ before standardization and reapplying it afterward to maintain its intended influence on the CPM.

Request for Feedback

Does this approach seem valid, or am I overlooking something? Are there alternative methods or considerations that might better address the issue of sign changes during standardization in the context of the CPM?

Answer 1

You are focussing way too much on the sign; and yes, that is misleading you.

You should only worry about the order or direction. Ask yourself: will a model with better performance get a higher or a lower metric. This order / direction does not change with the standardization.

So: Just keep your CPM as intended.