# Tag Info

3

A negative value means you're getting a terrible fit - which makes sense if you create a test set that doesn't have the same distribution as the training set. From the sklearn documentation: The coefficient $R^2$ is defined as (1 - u/v), where u is the residual sum of squares ((y_true - y_pred) ** 2).sum() and v is the total sum of squares ((y_true - ...

3

When you have a linear regression (without any scaling, just plain numbers) and you have a model with one explanatory variable $x$ and coefficients $\beta_0=0$ and $\beta_1=1$, then you essentially have a (estimated) function: $$y = 0 + 1x .$$ This tells you that when $x$ goes up (down) by one unit, $y$ goes up (down) by one unit. In this case it is just a ...

3

Regularization of Ridge causes its weight to become very close to zero, but not zero. In contrast lasso can make weights equal to zero because of the types of regularization they use. I would recommend to use lasso if you have lots of features, and you think just few of them are important. Otherwise, even though your model becomes simpler you will have a ...

3

Both formulations lead to the same solution if you correctly choose $C$ for both cost functions and if $C>0$. If we have the regularized loss $$J_1=\dfrac{1}{2}\sum_{n=1}^Ne_n^2+\dfrac{1}{2}C\sum_{k=0}^pw_k^2$$ we will have strong regularization for larger $C$ and small regularization for small positive $C$. If we devide the loss $J_1$ by the positive ...

3

Unlike lasso, ridge does not have zeroing coefficients as a goal, and you shouldn't expect applying ridge penalty to have this effect. So the answer to your title question is "no." However, in your question body, you ask whether it is possible for the ridge penalty to produce a zero coefficient that was nonzero in an unpenalized solution. The answer here ...

2

As mentioned by others, only Lasso can shrink parameters to exactly zero (while Ridge or Elastic Net will not, see Ch. 6.2.2 in Introduction to Statistical Learning). Lasso has some advantages, i.e. it can be used to deal with high dimensional data. For feature selection, some use a "double Lasso" approach. In case you only want to do feature selection (or ...

2

As I know we have below equation for Ridge Regression: $$RSS_{Ridge} = \Sigma_{i=1}^{n} (\hat{y}_{i} - y_{i})^2 - \lambda \Sigma_{j=1}^{p}(\beta^2)$$ First of all, it seems to me, if lambda goes higher does not mean that coefficients go down with inverse relation to lambda. Because the power of beta is two and the lambda is one. ...

2

(To summarize the comment thread into an answer) Your original scores: Mean Absolute Error: 37216342513.01034 Root Mean Squared Error: 871869805169.7842 are based on the original-scale target variable and are between $10^{10}$ and $10^{12}$, at least significantly smaller than the mean of the features (and the target)? So these aren't automatically bad ...

2

When we implement penalized regression models we are saying that we are going to add a penalty to the sum of the squared errors. Recall that the sum of squared errors is the following and that we are trying to minimize this value with Least Squares Regression: $$SSE = \sum_{i=1}^{n}(y_i-\hat{y_i})^2$$ When the model overfits or there is collinearity present, ...

1

Expanding my comments into an answer. Ridge regression is a by definition an augmentation of Least Squares method, especialy for problems where the data may be highly correlated (there is what is called multi-colinearity). Lets assume the dependent variable is $y$ and $x_i$ are the independent variables. Then assume the true mapping between $X$ and $y$ is: ...

1

Kaggle is a crowd source platform with no quality control. It is to be expected that there will be deviations from best practices.

1

@Ethan is correct about the formulation of the lasso penalty, and I think it's particularly important to understand it in that form (for one thing, because that same penalty can work with other models like neural networks, tree models, generalized linear models, ...). But, to your question: If $\lambda=0.5$ then does it mean that those coefficients whose ...

1

These diagrams show the "constrained" version of lasso/ridge, in which you minimize the pure loss function subject to a constraint $\|\beta\|_1\leq t$ or $\|\beta\|_2\leq t$. (Another common version adds a penalty to the loss, and these are equivalent.) The bluish solid shapes are the set of points with $\|\beta\|\leq t$, on the left with L1 norm ...

1

I believe with scaling, the coeff. are scaled by the same level i.e. Std. Deviation times with Standardization and (Max-Min) times with Normalization If we look at all the features individually, we are basically shifting it and then scaling it down by a constant but $y$ is unchanged. So, if we imaging a line in a 2-D space, we are keeping the $y$ same and ...

1

With sklearn you can have two approaches for linear regression: 1) LinearRegression object uses Ordinary Least Squares (OLS) solver from scipy, as Learning rate (LR) is one of two classifiers which have closed form solution. This is achieve by just inverting and multiplicating some matrices. 2) SGDRegressor which is an implementation of stochastic gradient ...

1

It seems you are using a the standard scaler twice, once in your pipeline and once more in the TransformedTargetRegressor. Next to that, you are only fitting the scaler, never actually scaling the inputs (i.e. transforming the input).

1

I found my answer myself.. I'm not sure if I should delete the question, so I'll just share: Say for a given data point before normalising we have a*x + b*z = y Normalising makes this a * x/std(x) + b * z/std(z) = y This means that for standard deviations greater than 1 the features are shrunk and therefore the weights are blown up. In the code the y_hat ...

1

I am not going to much of your code as i can see you have only imported all the libraries: You are facing with Over-fitting issue, The below are few of things when i come accross the same situation: Build multiple models and check Goodness of fit and then implement. Cross-validation is something which you should look into to make sure you have choosen the ...

Only top voted, non community-wiki answers of a minimum length are eligible