3
$\begingroup$

I'm running a regression model on a pretty large data set and getting a fairly woeful $R^2$ score of ~0.2 (see plot below), despite the plot looking like the model is generally pointing in the right direction.

My question is, when you have over a million data points, how high can you realistically expect the $R^2$ to go in real world data with a decent amount of noise?

What prompts by scepticism of such traditional measures are articles such as this that discuss how quantity of data can degrade statistical tests.

Let me know what you think and any regression examples using $R^2$ score as a quality metric.

Predicted values vs. actual

Improve this question
$\endgroup$

3 Answers 3

Reset to default
4
$\begingroup$

The coefficient of determination $r^2$ is defined in terms of variance: it is the proportion of variance in the dependent variable that is explained by the independent variable. Variance is a property of normal distributed data. Hence, the coefficient of determination can only be used when you assume that both the dependent and independent variables are normally distributed.

Just like other properties of normal distributed data, the estimation of $r^2$ improves when the amount of data increases. With very little data a coincidental colinearity might be the case, but this is not possible with large amounts of data.

Back to your example. Your data is cleary not normal distributed, it has right-skew and has large outliers. For this reason it is not advisable to use $r^2$. Imagine for example that in the lower left corner (where the majority of the data is) you would observe a negative trend, but in general there is a positive trend. The regression line would be the same and the $r^2$ would be in the same range. This is known as the Simpsons paradox.

In short, if your data is normal distributed you can use $r^2$ for any size of dataset. If it is not normal distributed you cannot use $r^2$.

Improve this answer
$\endgroup$
4
  • $\begingroup$ An upvote for the nice answer Pieter, but what is the alternative method/test can be used here ? THat would useful, even for me, to learn what is the better suited test for non-normally distributed data (here target as far as I can tell? If you know, please edit your answer. $\endgroup$
    – TwinPenguins
    Feb 21, 2018 at 6:45
  • $\begingroup$ This is a great, informative answer thanks a lot. I guess an answer to Majid's question might be that in this case I'm am going to set an acceptable bound for the mean squared error of the regression, contextual to the problem I'm trying to solve, then work on getting the model to meet that criteria. $\endgroup$
    – jshep
    Feb 21, 2018 at 10:00
  • $\begingroup$ You might want to ask a new question if you want an elaborate answer. Quick things I can think of are log-transform and a rank-transform. $\endgroup$
    – Pieter
    Feb 21, 2018 at 11:49
  • $\begingroup$ Marginal normality is not an assumption of linear regression, where $R^2$ would apply. Why do you say it is not valid? $\endgroup$
    – Dave
    Mar 3, 2021 at 17:48
1
$\begingroup$

There is no general answer of what to expect for an $R^2$ score. And there is no general answer for whether a model with this $R^2$ score is a "good" model. There are many cases where (1) this kind of $R^2$ score is not unreasonable and (2) the model is still useful. Looking at this data, $R^2=0.243$ feels about right, with most of the data concentrated in an almost disk-like blob near the origin.

The article you link to deals with statistical confidence, which is a very different issue than $R^2$. I would guess that given the amount of data, despite a $R^2$ score you consider is low, the p-value will be very high. The article touches upon p-hacking, which is an issue if you consider many possible inputs with competing models. With just one $x$, as is here, if this is the only model you have built, this is reasonable.

Improve this answer
$\endgroup$
1
$\begingroup$

I think Paul's answer is a really good one. The one additional point I'd make about the $R^2$ score is that you should only compare different $R^2$ scores between models estimated on the same set of data. Conceptually, it does not make any sense to compare $R^2$ scores between models derived from different data because $R^2$ is itself just a measure of variance in the outcome explained by the model. Different sets of data will have different amounts of variance explainable.

This is a chief reason why defining a "good" $R^2$ score is hard. The $R^2$ score is of course dependent on your model's fit, but also on the data itself (and the set of data is dependent on its collection, as well as the domain from which it is drawn).

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.