Is there a definitive and more conclusive way of interpreting the R^2 score from a linear regression model in terms of prediction accuracy?

Question

I'm trying to find a definitive way to conclude the R^2 score from a prediction accuracy point of view rather than variance. How should I do it?

Conceptually, most blogs / articles explain R^2 as:

1. The variation in the data points that can be explained by the model.
2. A measure of how closely fitted the data points are to the regression line.

I find these answers to be unsatisfactory.

If the R^2 score is 0.7, is it right for me to say that 70% of the data has been predicted accurately by the model? Since most explanations are tied to the explanation of the variance I'm thinking that looking at it from an 'accuracy' stand point of view is wrong. It implies that if I predicted 10 random points on the x-axis I would accurately guess 7 of its actual values.

According to scikit-learn's documentation page at the 'score' section:

The coefficient R^2 is defined as (1 - u/v), where u is the residual sum of squares ((y_true - y_pred) ** 2).sum() and v is the total sum of squares ((y_true - y_true.mean()) ** 2).sum()

Apart from scikit-learn's way of calculation, another way of calculation is simply the squared of R (coefficient of correlation).

Which way is right? Are both the same? Or are they related in some way?

Assuming the right way of calculation is based off scikit-learn's documentation where the R^2 score is the variance from the predicted mean divided by the variance from the actual mean, is it even possible that the variance from the predicted mean is smaller than the variance from the actual mean?

How can the variance from the prediction model which is built upon another dataset (let's say dataset A) be smaller than the actual variance of dataset B?

Deeply appreciate any thoughts, comments or clarification.

Answer 1

Quoting Dr. Bruce Ratner:

"R-sq is a first-blush indicator of a good model. R-sq is often misused as the measure to assess which model produces better predictions...The root mean squared error (RMSE) is the measure for determining the better model. The smaller the RMSE value, the better the model is (the predictions are more precise)."

I sourced this from Dr. Ratner's LinkedIn, so I can't link to it. His website is: dmstat1.com

Be careful with R-sq. The more variables you use the higher your R-sq value will be no matter how good or bad your model is. If you are using more than a few predictor variables you should also look at Adjusted-R-sq.

If you would rather get a measure of accuracy you should look at RMSE. If you are comparing two models also look at Akaike Information Criterion (AIC) and Bayesian Information Criterion (BIC).

To answer your question, if the R-sq is 0.7, it is incorrect to say that 70% of the data has been predicted accurately by the model. R-sq is a measure of how much variance is accounted for within the predictor variables you are using. R-sq of 0.7 means that your predictor variables can explain 70% of the variance in the response variable.

For example pretend that the price of a cup of coffee is influenced ONLY by the price of water and the price of coffee beans. If I only know the price of water and I predict the price of coffee using only the price of water my R-sq is 0.5, because I know only 1/2 of the things that influence the price of a cup of coffee.

In reality, it is virtually impossible to know ALL predictors that influence an outcome (R-sq = 1) because we either don't know what variables influence an outcome, or the variables that influence an outcome aren't contained in the dataset.

Answer 2

Your first definition is correct. It's the fraction of variation explained by the model. It is not accuracy.

Your labels ("y" values) vary, and the model tries to explain why by predicting those values. Think of the variance as the sum of squared differences from their mean (variance is just the 1/n * total squared error). This is like the base case prediction in the case of no model, 0 coefficients, just an intercept, predicting the mean label for every data point. This is "v" in your quote, and the squared error of this 'null' model.

Consider your actual model, and how much the labels vary from these real predictions. The sum of those squares is the models error, it's "u" here, and it's the part the model still didn't get right. R2 = 1-u/v, or R2 = (v-u)/v, and this measures the fraction of all that variance in the labels (v) that is explained (v-u) by the model.

It falls out from the definition of correlation coefficient with some math; see Wikipedia.

R2 can be negative, but this means the model is worse than predicting just the mean of the labels, and is 'bad'. But applying some other model not fit on the data could sure do that. It could also be positive if the model was somewhat predictive.

It's not a measure of how many regressors you know either.