2
$\begingroup$

For a square matrix of random data, N columns and N rows. I am fitting two models, linear regression and Lasso.

For the linear regression, I achieve a perfect score in train set, while in the Lasso I achieve a score of 0.

import pandas as pd
import numpy as np
from sklearn import linear_model


N = 100
X = pd.DataFrame(np.random.rand(N,N))
y = np.random.randint(20, size=N)

lin = linear_model.LinearRegression().fit(X,y)
lasso = linear_model.Lasso().fit(X,y)

print('Linear regression score',lin.score(X,y))
print('Lasso score',lasso.score(X,y))

Linear regression score 1.0

Lasso score 0.0 My questions will be:

  1. Can someone give me an intuitive idea why for a square matrix, Lasso regression achieve different results from Linear regression?

  2. What is the intuition behind the Lasso shrinking?

Note: In this case, the matrix is squared and the data/target is created randomly.

$\endgroup$
5
$\begingroup$

A few things going on here:

  1. Your matrix is 100x100. So you have no degrees of freedom left in a linear model, which will cause $R^2=1$. See this post.

  2. You use random numbers. Thus, they should make little sense in terms of explaining your dependent variable (it's basically noise). Since Lasso "shrinks" parameters which are not useful (and none is useful given perfect noise), all parameters are set to zero. This in turn also gives you an $R^2=0$. In this case the model consists only of an intercept (should be equal to 0.5), which is essentially the mean.

Change the matrix to have less columns than rows and you will get a low (but positive) $R^2$ for linear regression. Also look at the Lasso coefficients, they are all zero.

Have a look at the predicted values as well. You will see that the linear model does not work well. So the R-squared gives no indication about model fit in this case.

You can also try to change the random pattern to have some trend and you will find that Lasso does not shrink all coefficients to zero.

import pandas as pd
import numpy as np
from sklearn import linear_model

n = 1000
k = 5

X = pd.DataFrame(np.random.rand(n,k))
y = np.random.randint(2, size=n)

lin = linear_model.LinearRegression().fit(X,y)
lasso = linear_model.Lasso().fit(X,y)

print(lin.coef_)
print(lin.intercept_)

print(lasso.coef_)
print(lasso.intercept_)

print('Linear regression score',lin.score(X,y))
print('Lasso score',lasso.score(X,y))

Output:

[-0.00446436 -0.04509905  0.07870413 -0.00165025 -0.03951546]
0.5010650787133537
[-0. -0.  0. -0. -0.]
0.495
Linear regression score 0.003094912043144493
Lasso score 0.0
| improve this answer | |
$\endgroup$
  • $\begingroup$ Hi! Thanks for the answer. I am looking particularly at the case of a square matrix, same columns and same rows. I am interested in understanding how does the shrinkage works even if for a linear model you have one more degree of liberty and you can fully fit the space. $\endgroup$ – Carlos Mougan Dec 28 '19 at 10:59
  • $\begingroup$ The use of the random numbers is done with purpose so there is no pattern in data and achieving any positive result is wrong. $\endgroup$ – Carlos Mougan Dec 28 '19 at 11:04
  • $\begingroup$ I believe that you are answering the question of how to avoid having $R^2=1$ is linear. Not what are the differences bt lasso and linear results for a square matrix. I am looking more for the intuition of shrinkage of the space. $\endgroup$ – Carlos Mougan Dec 28 '19 at 11:09
  • $\begingroup$ So then post a proper question and make clear what you ask for. I cannot read your mind $\endgroup$ – Peter Dec 28 '19 at 11:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.