# Tag Info

## Hot answers tagged lasso

6

A few things going on here: Your matrix is 100x100. So you have no degrees of freedom left in a linear model, which will cause $R^2=1$. See this post. You use random numbers. Thus, they should make little sense in terms of explaining your dependent variable (it's basically noise). Since Lasso "shrinks" parameters which are not useful (and none is useful ...

4

When p > n, the LASSO model can only sustain up to n variables (this can be proven using linear algebra, the rank of the data matrix in particular), leaving at least p - n variables out (some that might be predictive, consider a model where you use LASSO and all your variables are predictive). This is not necessairly a bad thing though, depending on your use ...

3

sklearn has such a functionality already for regression problems, in enet_path and lasso_path. There's an example notebook here. Those functions have some cython base to them, so are probably substantially faster than your version. One other improvement that you can include in your implementation without adding cython is to use "warm starts": ...

3

When you have a linear regression (without any scaling, just plain numbers) and you have a model with one explanatory variable $x$ and coefficients $\beta_0=0$ and $\beta_1=1$, then you essentially have a (estimated) function: $$y = 0 + 1x .$$ This tells you that when $x$ goes up (down) by one unit, $y$ goes up (down) by one unit. In this case it is just a ...

2

Standardizing/normalizing is generally the right thing to do, but it will make little/no difference with just one independent variable if you also adjust the regularization strength. With more than one independent variable, standardizing assures that the lasso's penalty applies more equally to all the variables. Without standardizing, a large-scale ...

2

Yes the L1 regularization will shrink the irrelevant feature coefficients to zero and hence it doesn't require feature selection. In fact it IS a commonly used feature selection technique. So basically you are performing feature selection!!

2

Interesting question. I'd say it is correct not to divide, due to the following reasoning... For linear regression there is no difference. The optimum of the cost function stays the same, regardless how it is scaled. When doing Ridge or Lasso, the division affects the relative importance between the least-squares and the regularization parts of the cost ...

2

When we implement penalized regression models we are saying that we are going to add a penalty to the sum of the squared errors. Recall that the sum of squared errors is the following and that we are trying to minimize this value with Least Squares Regression: $$SSE = \sum_{i=1}^{n}(y_i-\hat{y_i})^2$$ When the model overfits or there is collinearity present, ...

1

These diagrams show the "constrained" version of lasso/ridge, in which you minimize the pure loss function subject to a constraint $\|\beta\|_1\leq t$ or $\|\beta\|_2\leq t$. (Another common version adds a penalty to the loss, and these are equivalent.) The bluish solid shapes are the set of points with $\|\beta\|\leq t$, on the left with L1 norm ...

1

R^2 is a statistical measure of how close the data are to the fitted regression line. It does this by seeing percentage of the variance of dependent varible that's explained by independent variable. To know more about R^2 score refer this video. So, basically it is a metric to see how well model fits the data but it is not adequate. Refer this discussion on ...

1

I believe with scaling, the coeff. are scaled by the same level i.e. Std. Deviation times with Standardization and (Max-Min) times with Normalization If we look at all the features individually, we are basically shifting it and then scaling it down by a constant but $y$ is unchanged. So, if we imaging a line in a 2-D space, we are keeping the $y$ same and ...

1

Lasso does feature selection in the way that a penalty is added to the OLS loss function (see figure below). So you can say that features with low "impact" will be "shrunken" by the penalty term (you "regulate" the features). Because of the L1 penalty, the $\beta_i$ can become zero (which is not the case with Ridge, L2). In the ...

1

Kaggle is a crowd source platform with no quality control. It is to be expected that there will be deviations from best practices.

1

In answer to your first question: The reason that your RMSE proceeded to increase as you increased the strength of your regularization (the value of $\lambda$) can be explained by reviewing the intuition behind what is happening when you increase the regularization of your model. Why did could my RMSE have kept increasing as I increased my regularization ...

1

Standard Scalar trained on 30 features so it expects 30 features only. One simple hack you can do is, you can create a new Standard Scalar and train with those 20 features, and replace your pipeline Standard Scalar with the new one. For the LogisticRegression, get the non zero weights and set those weights to the new model with 20 features without any ...

1

Change (search over) the penalty parameter of lasso. FinalRevenue = RevenueSoFar is a good baseline "model," but hopefully your other features can improve on that. You might also consider just modeling the target FinalRevenue - RevenueSoFar.

1

Neural Networks are notoriously good at performance and bad at interpretability, i.e. it's very difficult (almost impossible) to explain why a particular prediction was made. It's even more difficult to link the prediction with the features, since the NN does a lot of intermediate calculations where all the features play a role. The notoriously good models ...

1

@Ethan is correct about the formulation of the lasso penalty, and I think it's particularly important to understand it in that form (for one thing, because that same penalty can work with other models like neural networks, tree models, generalized linear models, ...). But, to your question: If $\lambda=0.5$ then does it mean that those coefficients whose ...

1

With sklearn you can have two approaches for linear regression: 1) LinearRegression object uses Ordinary Least Squares (OLS) solver from scipy, as Learning rate (LR) is one of two classifiers which have closed form solution. This is achieve by just inverting and multiplicating some matrices. 2) SGDRegressor which is an implementation of stochastic gradient ...

1

Stacking is going to help most when individual models capture unique characteristics of the data. It is often the case that different architectures perform similarly, if somewhat differently, on the same data. In those cases, ensembling/stacking will only offer slight incremental benefits. In the limit, of you only care about prediction, you can wire up as ...

1

Lambda is a tuning parameter („how much regularisation“, I think called alpha in sklearn) and you would choose lambda so that you optimise fit (e.g. by MSE). You can do this by running cross validation. This page (for the GLMnet package in R) explains how to apply Lasso in a very instructive way (alpha is the elastic-net mixing parameter here, Lambda is ...

1

Here is a little lasso example using the Boston Housing Data. The code also shows how to: choose optimal alpha, display data and predictions, display estimated coefficients, and how to make a prediction by hand. # Import toy data from sklearn.datasets import load_boston import pandas as pd # Load toy data boston = load_boston(return_X_y=False) # Make ...

1

Lasso() created a Lasso regressor object. The .fit() argument makes the regressor "learn" from your data. The parameters are tuned in order to "fit" the training data. Once the regressor object is fit, you can use it to .predict() on test data.

Only top voted, non community-wiki answers of a minimum length are eligible