why R-square always keep increasing

Question

I have read in multiple articles that R-square always increases with the number of features, even though a feature may not be of any significance.

The formula for R-square is

$$1 - \frac{\sum(y-\hat{y})^2}{\sum(y-\bar{y}^2)}$$

If the denominator is constant that means R-square is dependent upon only numerator, so basically on $\hat{y}$.

$$\hat{y} = a +b_1x_1 +b_2x_2 \ldots$$

Now if I have a new feature which is really not important shouldn't the beta coefficient of that feature be zero? And if it is really zero how will it really impact R-square?

Answer 1

What you are trying to avoid is including features that, while they do technically improve results on your sample data, they don't do a good job of generalizing to other hold-out sets. When you say "If I have a new feature which is really not important wont the beta coefficient of that feature should be zero" - you are correct that in this case it won't have an effect on R-squared. In the event that you include an unimportant feature and the coefficient is non-zero (meaning it's important on the sample data due to some random noise but not a true pattern in the underlying) then R-squared will increase and it will appear that you have a better model - but in fact you are leaning towards overfitting and you have a less robust model.

This point your article is making is pointing out a limitation of the R-squared evaluation criteria: if you add more degrees of freedom (input variables in this case) your score is likely only to go up (when maybe it shouldn't).

The F-test on the other hand recognizes this limitation of the R-squared and punishes the score by adding a degree of freedom term. So if you only see a marginal gain in R-square by adding a new term it will be punished moreso by the simple addition of the term (dof going up). See the following statquest for a good explanation (it gets to F-test towards the end of the video): https://www.youtube.com/watch?v=nk2CQITm_eo&ab_channel=StatQuestwithJoshStarmer

Answer 2

Linear regression has no way of knowing if features have significance or not. It will find the βs that produce the smallest squared error. That will usually not be zero even if the data is just noise. More features, regardless of significance, give more ways of describing the target variable and get a lower error.

Here is an example where you can see that this holds true even if all features have no significance at all:

from sklearn.linear_model import LinearRegression
from sklearn.metrics import r2_score
import numpy as np
import matplotlib.pyplot as plt

# generate data that is just noise
X = np.random.randn(100, 100)
y = np.random.randn(100, 1)

r2_scores = []

# fit 1-100 features on noise and calc r2 
for i in range(1, X.shape[1]+1):
    x = X[:,-i:]
    lr = LinearRegression()
    lr.fit(x.reshape(100, -1), y)
    coef_sum = lr.coef_.sum()
    r2 = r2_score(y, lr.predict(x.reshape(100, -1)))
    coefs.append(lr.coef_)
    r2_scores.append(r2)
    
plt.plot(r2_scores)
plt.xlabel('Number of features')
plt.ylabel('r2 score')

Which will give you something like: