Is it advisable to rerun LASSO multiple (2) times?

Question

I have a dataset with large (4500 variables) and 3000 observations (n < p case). I am using LASSO (from the glmnet package in R) to reduce dimensionality of the problem. I have couple of questions.

1) After I ran LASSO the first time, the number of variables with non-zero coefficients goes down to ~80 (which seems reasonable). But the correlation matrix shows that some of these variables are very highly correlated themselves (FORMS distinct groups if you plot it using the corplot package. So I ran LASSO exclusively on these variables whose coefficients were non-zero during the 1st run. This time, the algorithm picked up even less number of variables.I was expecting that they would remain more or less similar. I want to know if running LASSO multiple times is advisable or it introduces bias of any form in anyway that interferes with interpretation/prediction ?

2) My second question is about the magnitude of the coefficients. Some of the coefficients, in absolute value, turns out to be very close to 0 (~1E-14) while some are larger. I mean the spread is very large. Does it indicate that the variables with this small coefficients are making the model worse in any way. Is it advisable to get rid off those variables and rerun LASSO ?

Thoughts/comments are most welcome.

Thanks

Answer 1

Wow I'm surprised that no one has given this one a go. I'm very very late to the party but I'll attempt an answer just for information's sake.

Before I attempt to answer your questions I'd like to say that any situation in which the number of features exceeds the number of observations by 50% there may be some instability in the results.

What I'd recommend is to attempt a number of feature selection approaches and compare the results. Personally I'd start with a PCA to reduce the number of features such that n > p and go from there. But that's just me.

1) I'm not surprised by more estimates being zero at all. In the first lasso regression with this many features a large amount of these features are likely to be noise (no prediction power) meaning that any feature with a small capability of prediction would be selected in the first cut of variables. Running a secondary lasso regression might not be completely advisable (logically, can't really think of any theoretical issues at current) but it's likely that some of the stronger predictors are being uncovered on this second iteration.

2) The estimates in lasso regressions are quite difficult to interpret (apart from zero and non-zero values obviously), but what I'd recommend is to check whether you've scaled your features before running the lasso reg? As my understanding is that a lasso reg is a penalised version of the ridge regression, and the ridge regression needs the features to be scaled as otherwise features represented by larger numbers are given higher coefficients. If you have just ignore this, but if you want to interpret the estimates I'd switch to a Linear Reg/GLM not a lasso reg.

Hopefully this helps.