Questions tagged [regularization]
Inclusion of additional constraints (typically a penalty for complexity) in the model fitting process. Used to prevent overfitting / enhance predictive accuracy.
168 questions
2
votes
1
answer
31
views
Need advice regarding cross-validiation to obtain optimal lambda in Lasso
I am comparatively new to machine learning and any suggestions and coding corrections will be a great help.
I am using Lasso for feature selection and want to select the lambda that produces the ...
- 23
0
votes
1
answer
75
views
How to perform Backward Stepwise selection in Python
I am currently working through the book : An Introduction to Statistical Learning with Applications in Python. In the exercises so far I have been using the ISLP package.
I am currently trying to ...
user153827
1
vote
0
answers
6
views
Encouraging sparsity at block level or element-wise level?
I have an objective function $f(W)$, where $W$ is a $Kp \times Kp$ matrix. We can view $W$ is a $p \times p$ block matrix, where each block has the dimension $K \times K$. Now to optimize $f(W)$, I ...
- 11
2
votes
0
answers
25
views
What's the best way to incorporate momentum and regularization when training a neural network?
I want to implement the momentum algorithm to train a neural network, but I'm uncertain about where the regularization term should be incorporated. For ridge regularization, one option is to have:
$$
...
- 21
0
votes
0
answers
29
views
Implementing L2 Regularization in pure NumPy
I was implementing L2 Regularization in pure NumPy as an exercise to figure out the inner workings of the mathematics of an ML model.
I'm unsure if it's done well, I don't really have a reference to ...
- 11
0
votes
0
answers
33
views
Adaptive Lasso Coefficient Weights
I'm trying to understand how the Adaptive part of Adaptive Lasso works. I understand that theoretically, the weights for zero coefficients are inflated to infinity. But can someone explain this ...
0
votes
0
answers
24
views
Question about the limitations of regularization
I am training a neural network which is overfitting. Even when I increase the number of parameters, the test lost plateaus while the training loss keeps decreasing. Can regularization (like an L1 or ...
2
votes
1
answer
63
views
What does it mean if a neural networks starts overfitting more after applying regularisation techniques
Background
I am building a CNN to categorize cytometric cell data into healthy and diseased groups.
The architecture looks as follows:
3 Convolutional layers followed by average pooling followed by 3 ...
- 121
0
votes
1
answer
152
views
What may cause the CNN layer weight regularizer to reduce the model accuracy
What may cause the accuracy reduction when using the tf.keras regularizer at layers in CNN in the symptom? The example is simple but it happens with more complex CNN causing no improvement during the ...
- 791
0
votes
0
answers
49
views
Using Embedding For Regularization
Is using embeddings for regularization a valid practice?
My reasoning for that is that encoding training/tests datasets into smaller vectors would allow a smaller network with fewer parameters and ...
0
votes
1
answer
102
views
Linear Model With Highly Correlated Attributes Producing Inconsistent Weights
I know that having correlated attributes violates the linear model assumption of independent attributes, and I'm not interested in creating a more sophisticated model to tease apart the dependent ...
- 3
1
vote
0
answers
57
views
What is the l2-norm of a scalar
What is the meaning of the l2-norm when dealing with scalar values? I'm assuming it would be the same thing as taking the absolute value.
For context: I am trying to implement the clustering method ...
1
vote
1
answer
88
views
regularized LLS, trying to compute by hand the optimal weights yields wrong results
given the following dataset $S = \{(0,1),(1,1),(1,2)\}$ and the regularized problem
$$\sum_{i=1}^3 (y_i - w_1 x_i - w_0)^2 + \lambda w_1^2 \quad \lambda = 1 $$ i was tasked with finding the optimal $...
- 33
1
vote
1
answer
183
views
How can I implement some data constraints on a neural network?
I want to implement a NN model to predict the cost of my car trips. My dataset is something as follows (this is just a small sample):
...
- 123
0
votes
0
answers
27
views
Data augmentation layer based on physical model for time series data
I am quite new to the Keras API, so forgive me if I use incorrect terminology and for my lack of knowledge about the API.
This is for a mathematical (wave modelling) research project and I am quite ...
- 101
0
votes
1
answer
40
views
How to run a BE or FS Stepwise Regression on each dataset in a file folder full of datasets using lapply or map (without a loop)
All of the code in this question can be found in my GitHub Repository for this research project on Estimated Exhaustive Regression. Specifically, in the "Both BE & FS script" and "...
- 167
1
vote
1
answer
1k
views
Hyperparameter Tuning vs Regularization
While designing the architecture of a Neural Network, should I consider adding regularization (like Dropout, L1/L2, etc.) even after optimizing the problem using Hyperparameter Tuning? What should be ...
- 87
1
vote
1
answer
38
views
Test-set error in cross-validation with regularization
I am performing regularization (Ridge regression) using cross validation. I understand that for a certain value of the regularization parameter $\lambda$, we first fit on the training-set and then we ...
- 13
1
vote
1
answer
293
views
Why does feature selection matter if your model has L1 regularization?
I've been tinkering around with boosted trees, and I saw that for common libraries there is a parameter you can set to determine L1 regularization. I doubled my original feature set to around 130 ...
- 41
0
votes
1
answer
517
views
Do Linear Regression and Logistic Regression models from sklearn include regularization?
I'm learning Data Science by enrolling on different courses, and I've recently learnt something that seems very interesting to apply when doing linear or logistic regression models: regularization.
In ...
- 25
0
votes
1
answer
49
views
Gradient tree boosting additive training
In the XGBoost documentation, they specify that the additive training is done given an objective $obj^{(t)}$ defined as
$obj^{(t)} = \sum\limits_{i=1}^n \ell(y_i, \hat{y}_i^{(t-1)}+f_t(x_i)) + \sum\...
- 167
0
votes
1
answer
649
views
Why would we add regularization loss to the gradient itself in an SVM?
I'm doing CS 231n on my own. I'm looking at this solution to a question that implements a SVM.
Relevant code:
...
- 125
1
vote
0
answers
14
views
State-of-the-art techniques for regularizing Neural Networks?
For regularizing neural networks, I'm familiar with drop-out and l2/l1 regularization, which were the biggest players in the late 2010's.
Have any significant/strong competitors risen up since then?
- 111
1
vote
1
answer
97
views
Regularizing the intercept
I am reading The Elements of Statistical Learning and regarding regularized logistic regression it says:
"As with the lasso, we typically do not penalize the intercept term"
and I am ...
- 13
0
votes
1
answer
300
views
Regularization and loss function
I am currently trying to get a better understanding of regularization as a concept. This leads me to the following question:
Will regularization change when we change the loss function?
Is it correct ...
- 135
1
vote
1
answer
219
views
Is it possible to explain why Lasso models eliminated certain coefficient?
Is it possible to understand why Lasso models eliminated specific coefficients?. During the modelling, many of the highly correlated features in data is being eliminated by Lasso regression. Is it ...
- 263
3
votes
1
answer
52
views
Request: Confirmation on my understanding of overfitting and regularization concepts
Overfitted models tend to have largely different (some very high, some comparatively low) coefficients/weights for different feature values. So, this means the model (when drawn as graph) will have ...
- 161
5
votes
1
answer
294
views
Visualizing effect of regularization for linear regression problem
I wanted to put together an example notebook to demonstrate how regularization makes an impact for such a simple model as a simple linear regression. When executing the below script though, I notice ...
- 299
0
votes
1
answer
42
views
1
vote
1
answer
381
views
What is the purpose of positive parameter in sklearn.linear_model.ElasticNet?
I saw this parameter in the sklearn.linear_model.ElasticNet.
What is the purpose of this? What is the possible scenario where we want to force the coefficients to ...
- 263
1
vote
1
answer
84
views
Why are we not checking the significance of the coefficients in Lasso and elastic net models
As far as I know, we don't check the coefficient significance in Lasso and elasticnet models. Is it because insignificant feature coefficients will be driven to zero in these models?. Does that mean ...
- 263
1
vote
1
answer
556
views
Elegant way to plot the L2 regularization path of logistic regression in python?
Trying to plot the L2 regularization path of logistic regression with the following code (an example of regularization path can be found in page 65 of the ML textbook Elements of Statistical Learning ...
- 41
0
votes
1
answer
68
views
Do I have to remove features with pairwise correlation even if I am doing a regularized logistic regression?
Normally we would remove features that have high pairwise correlation with another feature before performing regression. But is this step necessary if I am applying L2 regularized logistic regression (...
- 41
2
votes
1
answer
154
views
What does "regularization" actually refer to?
I am familiar with regularization, where we add a penalty in our cost function to force the model to behave a certain way. But is this a definition of regularization?
Typically we regularize to get a &...
- 168
3
votes
1
answer
787
views
Why is l1 regularization rarely used comparing to l2 regularization in Deep Learning?
l1 regularization increases sparsity, so unimportant weights are decreased closer to 0. In Deep Learning models, the input usually consists of thousands or millions of features/pixels, and the network ...
- 131
1
vote
0
answers
177
views
Adding noise after LSTM layer
I am building a Natural Language Inference neural network model that learns to identify if one sentence (hypothesis) follows from another sentence (premise). So the input to my network is 2 sentences, ...
- 21
2
votes
0
answers
87
views
How do we bring Pareto optimality into the realm of Machine Learning?
I have a multi-objective optimisation problem with a large number of objectives (more than 10) which is generally the case in most-real life problems. The use of traditional GAs such as NSGA-II or ...
3
votes
1
answer
1k
views
Difference between PCA and regularisation
Currently, I am confusing about PCA and regularisation.
I wonder what is the difference between PCA and regularisation: particularly lasso (L1) regression?
Seems both of them can do the feature ...
- 133
1
vote
0
answers
33
views
Understanding model's learning curves
I'm trying to train a Lane Detection CNN called PINet on a proprietary dataset.
Below are some of the important configuration values:
Batch size: 6
Optimizer: Adam
Learning rate: High of 1e-4 and Low ...
- 11
-1
votes
1
answer
112
views
Lack of standardization in Kaggle jupyter notebooks when using lasso/ridge?
I've recently started using Kaggle, and I've noticed that for a lot of these jupyter notebooks written by others, when they use Ridge/Lasso, they don't standardize the non-categorical numerical ...
1
vote
1
answer
96
views
Why is the optimal C chosen by GridSearchCV so small?
I'm trying to use GridSearchCV to select the optimal C value in this simple SVM problem with non-separable samples. The issue I'm having is that when I run the code the optimal C is chosen to be ...
- 13
1
vote
0
answers
32
views
why use one regularisation technique over another?
why should I prefer L1 over L2, in fully-connected-layer or convolution?
why use dropout between 2 layers, when there is the option of regularising a layer(or both) with something like L1 or L2? and ...
2
votes
1
answer
54
views
Approximation of long sequence of layers by one layer
Consider the following situation : there is a deep neural network with a lot of layers, and in order to speed up the inference or for regularization purposes one would like to reduce the complexity of ...
1
vote
0
answers
621
views
Can elastic net l1 ratio be greater than 1?
I have multiple datasets that I trained with ElasticNetCV (sklearn), and I noticed that many of them selected l1_ratio = 1 as ...
- 221
1
vote
2
answers
896
views
What is the meaning of the sparsity parameter
Sparse methods such as LASSO contain a parameter $\lambda$ which is associated with the minimization of the $l_1$ norm. Higher the value of $\lambda$ ($>0$) means that more coefficients will be ...
- 541
0
votes
1
answer
74
views
Why use regularization?
In a linear model, regularization decreases the slope. Do we just assume that fitting a lin model on training data overfits by almost always creating a slope which is higher than it would be with ...
- 3
2
votes
0
answers
49
views
Entropy-regularized RL (G-learning) vs. IRL (Inverse Reinforcement Learning)
What are the differences between entropy-regularized RL (G-learning) and IRL (Inverse Reinforcement Learning)? and how are they applied to actual problems (besides stand-alone Markov decision ...
- 214
2
votes
1
answer
1k
views
Is it better to use separately regularization methods for Neural Networks (L2/L1 & Dropout)
I have been exploring different regularization approaches and observed the most common to be using either Dropout Layers or L1/L2 Regularization. I have seen many debates of whether it is of interest ...
3
votes
1
answer
2k
views
Relatively high regularization parameters for XGBoost model only way to prevent overfitting
I am modeling a continuous regression/forecasting problem for very right-skewed data. I've been using ElasticNet and Huber regression with quite a bit of success, and have recently moved into using ...
- 225
3
votes
2
answers
691
views
When I add regularization like L1,L2 , do I need more epochs to properly train my model?
When I add regularization techniques in my model like L1 or L2 do i need more epochs to properly converge my model.
...
- 709