I have a dataset with large (4500 variables) and 3000 observations (n < p case). I am using LASSO (from the glmnet package in R) to reduce dimensionality of the problem. I have couple of questions.
1) After I ran LASSO the first time, the number of variables with non-zero coefficients goes down to ~80 (which seems reasonable). But the correlation matrix shows that some of these variables are very highly correlated themselves (FORMS distinct groups if you plot it using the
corplot package. So I ran LASSO exclusively on these variables whose coefficients were non-zero during the 1st run. This time, the algorithm picked up even less number of variables.I was expecting that they would remain more or less similar. I want to know if running LASSO multiple times is advisable or it introduces bias of any form in anyway that interferes with interpretation/prediction ?
2) My second question is about the magnitude of the coefficients. Some of the coefficients, in absolute value, turns out to be very close to 0 (~1E-14) while some are larger. I mean the spread is very large. Does it indicate that the variables with this small coefficients are making the model worse in any way. Is it advisable to get rid off those variables and rerun LASSO ?
Thoughts/comments are most welcome.