Questions tagged [regularization]

Inclusion of additional constraints (typically a penalty for complexity) in the model fitting process. Used to prevent overfitting / enhance predictive accuracy.

170 questions
Filter by
Sorted by
Tagged with
14 views

What is regularization exactly? and when/why do we use it?

I have come across scenarios where and why regularization is useful: There are multiple Weight combinations where loss value attains minimum value (or almost close). To generalize better i.e to ...
• 186
15 views

How to perform Backward Stepwise selection in Python

I am currently working through the book : An Introduction to Statistical Learning with Applications in Python. In the exercises so far I have been using the ISLP package. I am currently trying to ...
1 vote
6 views

Encouraging sparsity at block level or element-wise level?

I have an objective function $f(W)$, where $W$ is a $Kp \times Kp$ matrix. We can view $W$ is a $p \times p$ block matrix, where each block has the dimension $K \times K$. Now to optimize $f(W)$, I ...
15 views

• 167
579 views

Why would we add regularization loss to the gradient itself in an SVM?

I'm doing CS 231n on my own. I'm looking at this solution to a question that implements a SVM. Relevant code: ...
• 125
1 vote
14 views

State-of-the-art techniques for regularizing Neural Networks?

For regularizing neural networks, I'm familiar with drop-out and l2/l1 regularization, which were the biggest players in the late 2010's. Have any significant/strong competitors risen up since then?
• 111
1 vote
83 views

Regularizing the intercept

I am reading The Elements of Statistical Learning and regarding regularized logistic regression it says: "As with the lasso, we typically do not penalize the intercept term" and I am ...
267 views

Regularization and loss function

I am currently trying to get a better understanding of regularization as a concept. This leads me to the following question: Will regularization change when we change the loss function? Is it correct ...
• 135
1 vote
190 views

Is it possible to explain why Lasso models eliminated certain coefficient?

Is it possible to understand why Lasso models eliminated specific coefficients?. During the modelling, many of the highly correlated features in data is being eliminated by Lasso regression. Is it ...
• 263
50 views

Request: Confirmation on my understanding of overfitting and regularization concepts

Overfitted models tend to have largely different (some very high, some comparatively low) coefficients/weights for different feature values. So, this means the model (when drawn as graph) will have ...
• 161
290 views

Visualizing effect of regularization for linear regression problem

I wanted to put together an example notebook to demonstrate how regularization makes an impact for such a simple model as a simple linear regression. When executing the below script though, I notice ...
• 299
42 views

** ...
1 vote
359 views

What is the purpose of positive parameter in sklearn.linear_model.ElasticNet?

I saw this parameter in the sklearn.linear_model.ElasticNet. What is the purpose of this? What is the possible scenario where we want to force the coefficients to ...
• 263
1 vote
72 views

Why are we not checking the significance of the coefficients in Lasso and elastic net models

As far as I know, we don't check the coefficient significance in Lasso and elasticnet models. Is it because insignificant feature coefficients will be driven to zero in these models?. Does that mean ...
• 263
1 vote
532 views

Elegant way to plot the L2 regularization path of logistic regression in python?

Trying to plot the L2 regularization path of logistic regression with the following code (an example of regularization path can be found in page 65 of the ML textbook Elements of Statistical Learning ...
62 views

Do I have to remove features with pairwise correlation even if I am doing a regularized logistic regression?

Normally we would remove features that have high pairwise correlation with another feature before performing regression. But is this step necessary if I am applying L2 regularized logistic regression (...
148 views

What does "regularization" actually refer to?

I am familiar with regularization, where we add a penalty in our cost function to force the model to behave a certain way. But is this a definition of regularization? Typically we regularize to get a &...
• 168
717 views

Why is l1 regularization rarely used comparing to l2 regularization in Deep Learning?

l1 regularization increases sparsity, so unimportant weights are decreased closer to 0. In Deep Learning models, the input usually consists of thousands or millions of features/pixels, and the network ...
• 131
1 vote
174 views

I am building a Natural Language Inference neural network model that learns to identify if one sentence (hypothesis) follows from another sentence (premise). So the input to my network is 2 sentences, ...
• 21
83 views

How do we bring Pareto optimality into the realm of Machine Learning?

I have a multi-objective optimisation problem with a large number of objectives (more than 10) which is generally the case in most-real life problems. The use of traditional GAs such as NSGA-II or ...
904 views

Difference between PCA and regularisation

Currently, I am confusing about PCA and regularisation. I wonder what is the difference between PCA and regularisation: particularly lasso (L1) regression? Seems both of them can do the feature ...
• 133
1 vote
32 views

Understanding model's learning curves

I'm trying to train a Lane Detection CNN called PINet on a proprietary dataset. Below are some of the important configuration values: Batch size: 6 Optimizer: Adam Learning rate: High of 1e-4 and Low ...
• 11
111 views

Lack of standardization in Kaggle jupyter notebooks when using lasso/ridge?

I've recently started using Kaggle, and I've noticed that for a lot of these jupyter notebooks written by others, when they use Ridge/Lasso, they don't standardize the non-categorical numerical ...
1 vote
93 views

Why is the optimal C chosen by GridSearchCV so small?

I'm trying to use GridSearchCV to select the optimal C value in this simple SVM problem with non-separable samples. The issue I'm having is that when I run the code the optimal C is chosen to be ...
• 13
1 vote
30 views

why use one regularisation technique over another?

why should I prefer L1 over L2, in fully-connected-layer or convolution? why use dropout between 2 layers, when there is the option of regularising a layer(or both) with something like L1 or L2? and ...
53 views

Approximation of long sequence of layers by one layer

Consider the following situation : there is a deep neural network with a lot of layers, and in order to speed up the inference or for regularization purposes one would like to reduce the complexity of ...
1 vote
603 views

Can elastic net l1 ratio be greater than 1?

I have multiple datasets that I trained with ElasticNetCV (sklearn), and I noticed that many of them selected l1_ratio = 1 as ...
• 221
1 vote
840 views

What is the meaning of the sparsity parameter

Sparse methods such as LASSO contain a parameter $\lambda$ which is associated with the minimization of the $l_1$ norm. Higher the value of $\lambda$ ($>0$) means that more coefficients will be ...
• 541
72 views

Why use regularization?

In a linear model, regularization decreases the slope. Do we just assume that fitting a lin model on training data overfits by almost always creating a slope which is higher than it would be with ...