Hot answers tagged

6 votes

For a square matrix of data, I achieve $R^2=1$ for Linear Regression and $R^2=0$ for Lasso. What's the intuition behind?

A few things going on here: Your matrix is 100x100. So you have no degrees of freedom left in a linear model, which will cause $R^2=1$. See this post. You use random numbers. Thus, they should make ...
user avatar
  • 6,912
4 votes

Why does Lasso behave "erratically" when the number of features is greater than the number of training instances?

When p > n, the LASSO model can only sustain up to n variables (this can be proven using linear algebra, the rank of the data matrix in particular), leaving at least p - n variables out (some that ...
user avatar
  • 1,834
3 votes
Accepted

Elegant way to plot the L2 regularization path of logistic regression in python?

sklearn has such a functionality already for regression problems, in enet_path and lasso_path. There's an example notebook here....
user avatar
  • 9,698
3 votes

How do standardization and normalization impact the coefficients of linear models?

When you have a linear regression (without any scaling, just plain numbers) and you have a model with one explanatory variable $x$ and coefficients $\beta_0=0$ and $\beta_1=1$, then you essentially ...
user avatar
  • 6,912
2 votes

Normalisation results in R^2 score of 0 - Lasso regression

Standardizing/normalizing is generally the right thing to do, but it will make little/no difference with just one independent variable if you also adjust the regularization strength. With more than ...
user avatar
  • 9,698
2 votes
Accepted

What is the meaning of the sparsity parameter

When we implement penalized regression models we are saying that we are going to add a penalty to the sum of the squared errors. Recall that the sum of squared errors is the following and that we are ...
user avatar
  • 1,411
2 votes
Accepted

Difference between PCA and regularisation

Lasso does feature selection in the way that a penalty is added to the OLS loss function (see figure below). So you can say that features with low "impact" will be "shrunken" by ...
user avatar
  • 6,912
2 votes
Accepted

Do I have to remove features with pairwise correlation even if I am doing a regularized logistic regression?

Yes the L1 regularization will shrink the irrelevant feature coefficients to zero and hence it doesn't require feature selection. In fact it IS a commonly used feature selection technique. So ...
user avatar
  • 1,263
2 votes
Accepted

What's the correct cost function for Linear Regression

Interesting question. I'd say it is correct not to divide, due to the following reasoning... For linear regression there is no difference. The optimum of the cost function stays the same, regardless ...
user avatar
  • 266
1 vote
Accepted

Why is gridsearchCV.best_estimator_.score giving me r2_score even if I mentioned MAE as my main scoring metric?

This is the default behavior for any Scikit-learn regressor, and as far as I know, it cannot be modified. So for regressors, the score method will return the $R^2$ ...
user avatar
  • 2,191
1 vote

Is it possible to explain why Lasso models eliminated certain coefficient?

Have a look at "Introduction to Statistical Learning" (Chapter 6.2.2). The Lasso adds an aditional penalty term to the original OLS penalty. In addition to the residual sum of squares (RSS, ...
user avatar
  • 6,912
1 vote
Accepted

Accessing regression coefficients when using MultiOutputRegressor

Instead of using the estimator attribute you should be using the best_estimator attribute, after which you can access the ...
user avatar
  • 5,989
1 vote

How to set coefficient limit in lasso regression in Python?

Scikit-learn (which I'm assuming you're using) does not allow you to constrain the coefficients in such a way (at most you can constrain them to all be positive with ...
user avatar
1 vote

Lack of standardization in Kaggle jupyter notebooks when using lasso/ridge?

Kaggle is a crowd source platform with no quality control. It is to be expected that there will be deviations from best practices.
user avatar
1 vote

Lasso regression not getting better without random features

In answer to your first question: The reason that your RMSE proceeded to increase as you increased the strength of your regularization (the value of $\lambda$) can be explained by reviewing the ...
user avatar
  • 1,411
1 vote

How to remove features from a sklearn pipeline after it has already been fitted?

Standard Scalar trained on 30 features so it expects 30 features only. One simple hack you can do is, you can create a new ...
user avatar
  • 516
1 vote
Accepted

Lasso Regression for Feature Importance saying almost every feature is unimportant?

Change (search over) the penalty parameter of lasso. FinalRevenue = RevenueSoFar is a good baseline "model," but hopefully your other features can ...
user avatar
  • 9,698
1 vote

Interpreting machine learning coefficients

Neural Networks are notoriously good at performance and bad at interpretability, i.e. it's very difficult (almost impossible) to explain why a particular prediction was made. It's even more difficult ...
user avatar
  • 22.1k
1 vote

What is the meaning of the sparsity parameter

@Ethan is correct about the formulation of the lasso penalty, and I think it's particularly important to understand it in that form (for one thing, because that same penalty can work with other models ...
user avatar
  • 9,698
1 vote

how Lasso regression helps to shrinks the coefficient to zero and why ridge regression dose not shrink the coefficient to zero?

These diagrams show the "constrained" version of lasso/ridge, in which you minimize the pure loss function subject to a constraint $\|\beta\|_1\leq t$ or $\|\beta\|_2\leq t$. (Another ...
user avatar
  • 9,698
1 vote

regarding lasso.score in lasso modeling using scikit-learn

R^2 is a statistical measure of how close the data are to the fitted regression line. It does this by seeing percentage of the variance of dependent varible that's explained by independent variable. ...
user avatar
  • 1,366
1 vote

How do standardization and normalization impact the coefficients of linear models?

I believe with scaling, the coeff. are scaled by the same level i.e. Std. Deviation times with Standardization and (Max-Min) times with Normalization If we look at all the features individually, we ...
user avatar
  • 5,184
1 vote
Accepted

How is learning rate calculated in sklearn Lasso regression?

With sklearn you can have two approaches for linear regression: 1) LinearRegression object uses Ordinary Least Squares (OLS) solver from scipy, as Learning rate (...
user avatar
1 vote
Accepted

When should we start using stacking of models?

Stacking is going to help most when individual models capture unique characteristics of the data. It is often the case that different architectures perform similarly, if somewhat differently, on the ...
user avatar
  • 901
1 vote

LASSO remaining features for different penalisation

Lambda is a tuning parameter („how much regularisation“, I think called alpha in sklearn) and you would choose lambda so that you optimise fit (e.g. by MSE). You can do this by running cross ...
user avatar
  • 6,912
1 vote
Accepted

What happens when scikit-learn does a Lasso Model?

Here is a little lasso example using the Boston Housing Data. The code also shows how to: choose optimal alpha, display data and predictions, display estimated coefficients, and how to make a ...
user avatar
  • 6,912
1 vote

What happens when scikit-learn does a Lasso Model?

Lasso() created a Lasso regressor object. The .fit() argument makes the regressor "learn" from your data. The parameters are ...
user avatar
  • 5,687

Only top scored, non community-wiki answers of a minimum length are eligible