2
$\begingroup$

I have my confusion matrix as C.mat

8263    20    39
2       3826  14
43       7    4431 

My predicted class labels are Ypred and actual labels are Ytest. Ypred size is 16000*1 and Ytest 16000*1.

I am trying to calculate the R-squared and RMSE. Is there a way to directly calculate RMSE and R-squared from the confusion matrix?

I tried this:

RMSE = sqrt(immse(Ypred, Ytest))

However, it didn't work.

I can use either R or Matlab.

Any advice will be appreciated!

$\endgroup$
1
$\begingroup$

The confusion matrix suggests that you are performing classification rather than regression. RMSE and R-square are measures associated with continuous variables; For categorical variables, I'd suggest using Accuracy / Recall / Precision / F1 score to measure the performance of the model.

https://www.quora.com/How-is-root-mean-square-error-RMSE-and-classification-related

$\endgroup$
  • $\begingroup$ yes! i can get these accuracy recall scores from matrix but i am looking to get RMSE and R square as added metrics. Does this mean that there isnt a way to get Rsquare and RMSE values from Ypred and Ytest $\endgroup$ – sam venu Dec 5 '16 at 1:41
  • 1
    $\begingroup$ Even if we use one-hot encoding to encode the 3-class variable and obtain 2 binary variables to represent the 3 groups, I don't see how we could compute the R-Square / RMSE. If you consider the error of Ypred <> Ytest = 1, then RMSE is the same as accuracy isn't it? :) Why are you after RMSE / R-square for categorical target variable? $\endgroup$ – jkyh Dec 5 '16 at 1:46
1
$\begingroup$

In addition to what the other respondents said, I would like to add that using RMSE and MSE as metrics to evaluate a classifier can actually be a good idea if the classes are ordinal. In this case, there is a natural order between the categories, i.e. good > moderate > poor. Because missing by one class is less bad than missing by two or more, you want to use a metric that takes this into account. If you want to use only a single metric, MSE and MAE are your best choices according to Gaudette and Japkowicz (2009). Judging from their approach, RMSE could also be a good choice.

Here is one way to calculate MSE and RMSE from a confusion matrix in MATLAB:

cm = [8263   20   39
         2 3826   14
        43    7 4431];

se = 0;
for i = 1:3
    for j = 1:3
        se = se + cm(i,j) * (i-j)^2;
    end
end

mse = se / sum(sum(cm));
rmse = sqrt(mse);
$\endgroup$
0
$\begingroup$

If you use RMSE for classification, then effectively every squared error will be a 1. The mean squared error will be your misclassification rate and the RMSE the square root of that.

Other Helpful Information

The square root of the mean/average of the square of all of the error.

The use of RMSE is very common and it makes an excellent general purpose error metric for numerical predictions.

Compared to the similar Mean Absolute Error, RMSE amplifies and severely punishes large errors.

$$RMSE = \sqrt{\frac{1}{n}\sum_{i=1}^{n}((y_{i} - \hat{y}_{i})^2)}$$

MATLAB Code

RMSE = sqrt(mean((y-y_pred).^2));

R Code

RMSE <- sqrt(mean((y-y_pred)^2))

Python (Scikit Learn)

from sklearn.metrics import mean_squared_error
RMSE = mean_squared_error(y, y_pred)**0.5
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.