# Which metric to use for evaluating a rating system

I have a system which gives a star rating of the quality of work(on scale of 1-5, 1 being extremely poor and 5 being exceptionally good). An expert labelled a test set with their ratings of quality of work.

The data is imbalanced in the 5 classes(1,2,3,4,5), very few samples for Extremely poor and Exceptionally good ratings. Is there a metric which can take into account consideration that if system rated extremely poor quality work as very good or exceptionally good, it should be penalised more, as compared to when it rated an extremely poor rating as poor or average? And again due to imbalanced class distribution, straightforward accuracy is not the best measure.

Any suggestions for metric?

Performance Measures for Multi-Class Problems

For classification problems, classifier performance is typically defined according to the confusion matrix associated with the classifier. Based on the entries of the matrix, it is possible to compute sensitivity (recall), specificity, and precision. For a single cutoff, these quantities lead to balanced accuracy (sensitivity and specificity) or to the F1-score (recall and precision). To evaluate a scoring classifier at multiple cutoffs, these quantities can be used to determine the area under the ROC curve (AUC) or the area under the precision-recall curve (AUCPR).

All of these performance measures are easily obtainable for binary classification problems. Which measure is appropriate depends on the type of classifier.

F-Measure/F1-Score for Imbalanced Classification

Classification accuracy is widely used because it is one single measure used to summarize model performance. F-Score provides a way to combine both precision and recall into a single measure that captures both properties. Alone, neither precision nor recall tells the whole story. We can have excellent precision with terrible recall, or alternately, terrible precision with excellent recall. F-measure provides a way to express both concerns with a single score. Once precision and recall have been calculated for a binary or multiclass classification problem, the two scores can be combined into the calculation of the F-Measure. The traditional F measure is calculated as follows:

$$\begin{equation} \nonumber F-Measure=\frac{2 * Precision * Recall}{Precision + Recall} \end{equation}$$

For a multi-class classification problem, we don’t calculate an overall F-1 score. Instead, we calculate the F-1 score per class in a one-vs-rest manner.

Here are simple Python functions that do exactly that:

def precision(y_true, y_pred):
i = set(y_true).intersection(y_pred)
len1 = len(y_pred)
if len1 == 0:
return 0
else:
return len(i) / len1

def recall(y_true, y_pred):
i = set(y_true).intersection(y_pred)
return len(i) / len(y_true)

def f1(y_true, y_pred):
p = precision(y_true, y_pred)
r = recall(y_true, y_pred)
if p + r == 0:
return 0
else:
return 2 * (p * r) / (p + r)


For imbalanced classes, I would suggest to go with Weighted F1-Score or Average AUC/Weighted AUC or Micro and macro averages of the F1-score

Weighted F1-Score was calculated by F1 score for each class independently but when it adds them together uses a weight that depends on the number of true labels of each class:

$$F1_{class1}*W_1+F1_{class2}*W_2+\cdot\cdot\cdot+F1_{classN}*W_N$$

therefore favoring the majority class.

Micro and macro averages of the F1-score

Micro and macro averages represent two ways of interpreting confusion matrices in multi-class settings. Here, we need to compute a confusion matrix for every class $$g_i \in G = \{1, \ldots, K\}$$ such that the $$i-th$$ confusion matrix considers class $$g_i$$ as the positive class and all other classes $$g_j$$ with $$j≠i$$ as the negative class. Since each confusion matrix pools all observations labeled with a class other than $$g_i$$ as the negative class, this approach leads to an increase in the number of true negatives, especially if there are many classes.

To exemplify why the increase in true negatives is problematic, imagine there are 10 classes with 10 observations each. Then the confusion matrix for one of the classes may have the following structure:

Prediction/Reference    Class 1     Other Class
Class 1                   8             10
Other Class               2             80


Based on this matrix, the specificity would be $$\frac{80}{80 + 10} = 88.9\%$$ although class 1 was only correctly predicted in 8 out of 18 instances (precision 44.4%). Thus, since the negative class is predominant, the specificity becomes inflated. Thus, micro- and macro averages are only defined for the F1-score and not for the balanced accuracy, which relies on the true negative rate.

In the following, we will use $$TP_i$$, $$FP_i$$, and $$FN_i$$ to respectively indicate true positives, false positives, and false negatives in the confusion matrix associated with the $$i-th$$ class. Moreover, let precision be indicated by $$P$$ and recall by $$R$$.

The micro average has its name from the fact that it pools the performance over the smallest possible unit (i.e. over all samples):

\begin{align*} P_{\rm{micro}} &= \frac{\sum_{i=1}^{|G|} TP_i}{\sum_{i=1}^{|G|} TP_i+FP_i} \\ R_{\rm{micro}} &= \frac{\sum_{i=1}^{|G|} TP_i}{\sum_{i=1}^{|G|} TP_i + FN_i} \end{align*}

The micro-averaged precision, $$P_{micro}$$, and recall, $$R_{micro}$$, give rise to the micro F1-score: $$F1_{\rm{micro}} = 2 \frac{P_{\rm{micro}} \cdot R_{\rm{micro}}}{P_{\rm{micro}} + R_{\rm{micro}}}$$

If a classifier obtains a large $$F1_{micro}$$, this indicates that it performs well overall. The micro-average is not sensitive to the predictive performance for individual classes. As a consequence, the micro-average can be particularly misleading when the class distribution is imbalanced.

The macro average has its name from the fact that it averages over larger groups, namely over the performance for individual classes rather than observations:

\begin{align*} P_{\rm{macro}} &= \frac{1}{|G|} \sum_{i=1}^{|G|} \frac{TP_i}{TP_i+FP_i} = \frac{\sum_{i=1}^{|G|} P_i}{|G|}\\ R_{\rm{macro}} &= \frac{1}{|G|} \sum_{i=1}^{|G|} \frac{TP_i}{TP_i + FN_i} = \frac{\sum_{i=1}^{|G|} R_i}{|G|} \end{align*}

The macro-averaged precision and recall give rise to the macro F1-score:

$$F1_{\rm{macro}} = 2 \frac{P_{\rm{macro}} \cdot R_{\rm{macro}}}{P_{\rm{macro}} + R_{\rm{macro}}}$$

If $$F1_{macro}$$ has a large value, this indicates that a classifier performs well for each individual class. The macro-average is, therefore, more suitable for data with an imbalanced class distribution.

References:

• Thanks for your detailed response. But all these metrics consider outputs as categorical values with a rating of 5 ,if labelled as 4 as wrong as if it was labelled 1, which is not my requirement. Let's put Machine Learning aside for a while and not think of this as a classification problem. What is this is a rule based system? I can define a custom function for this, but I feel sure people would have encountered this in past as well, am I missing something basic? Even for 5 class fine-grained sentiment analysis, what metric will tell me if system rates 4 as 5, its acceptable but 1/2 isn't.
– Jazz
Jan 25 at 21:32