Calculating RMSE AND R-squared from the confusion matrix

Question

I have my confusion matrix as C.mat

8263    20    39
2       3826  14
43       7    4431 

My predicted class labels are Ypred and actual labels are Ytest. Ypred size is 16000*1 and Ytest 16000*1.

I am trying to calculate the R-squared and RMSE. Is there a way to directly calculate RMSE and R-squared from the confusion matrix?

I tried this:

RMSE = sqrt(immse(Ypred, Ytest))

However, it didn't work.

I can use either R or Matlab.

Any advice will be appreciated!

Answer 1

The confusion matrix suggests that you are performing classification rather than regression. RMSE and R-square are measures associated with continuous variables; For categorical variables, I'd suggest using Accuracy / Recall / Precision / F1 score to measure the performance of the model.

https://www.quora.com/How-is-root-mean-square-error-RMSE-and-classification-related

Answer 2

If you use RMSE for classification, then effectively every squared error will be a 1. The mean squared error will be your misclassification rate and the RMSE the square root of that.

Other Helpful Information

The square root of the mean/average of the square of all of the error.

The use of RMSE is very common and it makes an excellent general purpose error metric for numerical predictions.

Compared to the similar Mean Absolute Error, RMSE amplifies and severely punishes large errors.

$$RMSE = \sqrt{\frac{1}{n}\sum_{i=1}^{n}((y_{i} - \hat{y}_{i})^2)}$$

MATLAB Code

RMSE = sqrt(mean((y-y_pred).^2));

R Code

RMSE <- sqrt(mean((y-y_pred)^2))

Python (Scikit Learn)

from sklearn.metrics import mean_squared_error
RMSE = mean_squared_error(y, y_pred)**0.5

Answer 3

In addition to what the other respondents said, I would like to add that using RMSE and MSE as metrics to evaluate a classifier can actually be a good idea if the classes are ordinal. In this case, there is a natural order between the categories, i.e. good > moderate > poor. Because missing by one class is less bad than missing by two or more, you want to use a metric that takes this into account. If you want to use only a single metric, MSE and MAE are your best choices according to Gaudette and Japkowicz (2009). Judging from their approach, RMSE could also be a good choice.

Here is one way to calculate MSE and RMSE from a confusion matrix in MATLAB:

cm = [8263   20   39
         2 3826   14
        43    7 4431];

se = 0;
for i = 1:3
    for j = 1:3
        se = se + cm(i,j) * (i-j)^2;
    end
end

mse = se / sum(sum(cm));
rmse = sqrt(mse);