Interpreting ROC Scores for non-binary data

Question

I have been handed a random-forest based machine learning model that is predicting a rate rather than a binary category (e.g. the rate of road quality deterioration in a city for planning repaving requirements) and I need to understand how to evaluate the results. Most intro examples I can find rely on binary or multiple classes. How would you apply these concepts to a continuous rate? E.g. if the actual rate of deterioration is 2.78%/year and the model predicts 2.69%/yr vs 2.85%/yr?

1. How would you construct a confusion matrix for continuous data?
2. How would you interpret a ROC curve?

Answer 1

We have to distinguish different cases for prediction and actual outcome:

1. Categorical prediction and categorical outcome (special case: binary prediction & outcome)
2. Continuous predicition (e.g. probabilities) and categrical outcome (special case: binary outcome)
3. Continuous predicition and continuous outcome

All three require different evaluation methods. Number 1 and 2 are both classification problems (categorical outcome), while number 3 (your case) is a regression problem.

So how to evaluate these three cases:

1. Evaluating categorical prediction and categorical outcome

In this case one can count which combination of predicted class and actual class occurs how often. This allows to compute:

• Confusion matrix
• Accuracy
• Precision, Recall, Sensitivity, Specificity, ... (especially in the binary case)

In sklearn, such a prediction would be done by the predict-method of a Classifier.

2. Evaluating continuous prediction and categorical outcome

In this case, one only has a probability (or even more loose a weight), that indicates how likely a sample belongs to which class. There are now hard count, so a confusion matrix does not exists.

Typically, one uses the following:

• ROC-Curve / Area under the ROC-Curve (ROC-AUC)
• Precision-Recall-Curve (PR-AUC). This is especially interesting, if the number of true-negatives is not relevant.
• Loss, especially the cross.entropy-loss

In sklearn, such a prediction would be done by the predict_proba-method of a Classifier.

There are ways to transform this to a categorical predition, e.g.:

• Take the value with the highest probability / weight
• In the binary case: take a threshold and consider all values with probability / weight above that value as class 1

This allows to evaluate the categorical predictions

3. Evaluating continuous prediction and continuous outcome

Evaluating regression models is done differently. Typically, one measures how far the predictions are away from the actual outcome:

• Root-Mean-Squared_Error (RMSE) is a popular methods. It penalizes large difference strongly and can be interpreted in the scale / unit of the predition.
• $R^2$ measures how much better a predictor is than a constant prediction. Here 0% is the the performance of a constant mean value. 100% means the prediction is perfectly predicting the outcome.
• Mean-Absolute-Error (MAE) is similar to RMSE, but does not penalize large errors. Often, RMSE is preferable, because one wants to avoid totally wrong predictions.

In sklearn, such a prediction would be done by the predict-method of a Regressor.

If you really need to compute a confusion matrix, than you could discretize your outcomes (and predictions) into categories. Example:

• small: < 1.0%/yr
• medium:1.0%/yr - 2.0%/yr
• ...