Questions tagged [regularization]
Inclusion of additional constraints (typically a penalty for complexity) in the model fitting process. Used to prevent overfitting / enhance predictive accuracy.
170
questions
-1
votes
0
answers
14
views
What is regularization exactly? and when/why do we use it?
I have come across scenarios where and why regularization is useful:
There are multiple Weight combinations where loss value attains minimum value (or almost close).
To generalize better i.e to ...
- 186
0
votes
0
answers
15
views
How to perform Backward Stepwise selection in Python
I am currently working through the book : An Introduction to Statistical Learning with Applications in Python. In the exercises so far I have been using the ISLP package.
I am currently trying to ...
user153827
1
vote
0
answers
6
views
Encouraging sparsity at block level or element-wise level?
I have an objective function $f(W)$, where $W$ is a $Kp \times Kp$ matrix. We can view $W$ is a $p \times p$ block matrix, where each block has the dimension $K \times K$. Now to optimize $f(W)$, I ...
- 11
2
votes
0
answers
15
views
What's the best way to incorporate momentum and regularization when training a neural network?
I want to implement the momentum algorithm to train a neural network, but I'm uncertain about where the regularization term should be incorporated. For ridge regularization, one option is to have:
$$
...
- 21
0
votes
0
answers
20
views
Implementing L2 Regularization in pure NumPy
I was implementing L2 Regularization in pure NumPy as an exercise to figure out the inner workings of the mathematics of an ML model.
I'm unsure if it's done well, I don't really have a reference to ...
- 11
0
votes
0
answers
26
views
Adaptive Lasso Coefficient Weights
I'm trying to understand how the Adaptive part of Adaptive Lasso works. I understand that theoretically, the weights for zero coefficients are inflated to infinity. But can someone explain this ...
0
votes
0
answers
23
views
Question about the limitations of regularization
I am training a neural network which is overfitting. Even when I increase the number of parameters, the test lost plateaus while the training loss keeps decreasing. Can regularization (like an L1 or ...
2
votes
1
answer
56
views
What does it mean if a neural networks starts overfitting more after applying regularisation techniques
Background
I am building a CNN to categorize cytometric cell data into healthy and diseased groups.
The architecture looks as follows:
3 Convolutional layers followed by average pooling followed by 3 ...
- 121
0
votes
1
answer
150
views
What may cause the CNN layer weight regularizer to reduce the model accuracy
What may cause the accuracy reduction when using the tf.keras regularizer at layers in CNN in the symptom? The example is simple but it happens with more complex CNN causing no improvement during the ...
- 771
0
votes
0
answers
41
views
Using Embedding For Regularization
Is using embeddings for regularization a valid practice?
My reasoning for that is that encoding training/tests datasets into smaller vectors would allow a smaller network with fewer parameters and ...
-1
votes
0
answers
49
views
Need feedback on idea for new regularization term
I've been working on creating a regularization term that ensures that correlated attributes are given similar weights in a linear model. This helps to avoid some of the inconsistency in the weights of ...
- 1
0
votes
1
answer
95
views
Linear Model With Highly Correlated Attributes Producing Inconsistent Weights
I know that having correlated attributes violates the linear model assumption of independent attributes, and I'm not interested in creating a more sophisticated model to tease apart the dependent ...
- 1
-1
votes
0
answers
79
views
XGBoost Architecture Diagram required
Good Day!
My topic is general and theory related, about XGBoost working. I am searching XGBoost Architecture Diagram. I know it works on principles of Decision Trees, Bagging, Random Forest, Boosting, ...
- 99
1
vote
0
answers
51
views
What is the l2-norm of a scalar
What is the meaning of the l2-norm when dealing with scalar values? I'm assuming it would be the same thing as taking the absolute value.
For context: I am trying to implement the clustering method ...
1
vote
1
answer
88
views
regularized LLS, trying to compute by hand the optimal weights yields wrong results
given the following dataset $S = \{(0,1),(1,1),(1,2)\}$ and the regularized problem
$$\sum_{i=1}^3 (y_i - w_1 x_i - w_0)^2 + \lambda w_1^2 \quad \lambda = 1 $$ i was tasked with finding the optimal $...
- 33
1
vote
1
answer
151
views
How can I implement some data constraints on a neural network?
I want to implement a NN model to predict the cost of my car trips. My dataset is something as follows (this is just a small sample):
...
- 123
0
votes
0
answers
25
views
Data augmentation layer based on physical model for time series data
I am quite new to the Keras API, so forgive me if I use incorrect terminology and for my lack of knowledge about the API.
This is for a mathematical (wave modelling) research project and I am quite ...
- 101
0
votes
1
answer
39
views
How to run a BE or FS Stepwise Regression on each dataset in a file folder full of datasets using lapply or map (without a loop)
All of the code in this question can be found in my GitHub Repository for this research project on Estimated Exhaustive Regression. Specifically, in the "Both BE & FS script" and "...
- 167
1
vote
1
answer
1k
views
Hyperparameter Tuning vs Regularization
While designing the architecture of a Neural Network, should I consider adding regularization (like Dropout, L1/L2, etc.) even after optimizing the problem using Hyperparameter Tuning? What should be ...
- 87
1
vote
1
answer
37
views
Test-set error in cross-validation with regularization
I am performing regularization (Ridge regression) using cross validation. I understand that for a certain value of the regularization parameter $\lambda$, we first fit on the training-set and then we ...
- 13
1
vote
1
answer
229
views
Why does feature selection matter if your model has L1 regularization?
I've been tinkering around with boosted trees, and I saw that for common libraries there is a parameter you can set to determine L1 regularization. I doubled my original feature set to around 130 ...
- 41
0
votes
1
answer
423
views
Do Linear Regression and Logistic Regression models from sklearn include regularization?
I'm learning Data Science by enrolling on different courses, and I've recently learnt something that seems very interesting to apply when doing linear or logistic regression models: regularization.
In ...
- 25
0
votes
1
answer
47
views
Gradient tree boosting additive training
In the XGBoost documentation, they specify that the additive training is done given an objective $obj^{(t)}$ defined as
$obj^{(t)} = \sum\limits_{i=1}^n \ell(y_i, \hat{y}_i^{(t-1)}+f_t(x_i)) + \sum\...
- 167
0
votes
1
answer
579
views
Why would we add regularization loss to the gradient itself in an SVM?
I'm doing CS 231n on my own. I'm looking at this solution to a question that implements a SVM.
Relevant code:
...
- 125
1
vote
0
answers
14
views
State-of-the-art techniques for regularizing Neural Networks?
For regularizing neural networks, I'm familiar with drop-out and l2/l1 regularization, which were the biggest players in the late 2010's.
Have any significant/strong competitors risen up since then?
- 111
1
vote
1
answer
83
views
Regularizing the intercept
I am reading The Elements of Statistical Learning and regarding regularized logistic regression it says:
"As with the lasso, we typically do not penalize the intercept term"
and I am ...
- 13
0
votes
1
answer
267
views
Regularization and loss function
I am currently trying to get a better understanding of regularization as a concept. This leads me to the following question:
Will regularization change when we change the loss function?
Is it correct ...
- 135
1
vote
1
answer
190
views
Is it possible to explain why Lasso models eliminated certain coefficient?
Is it possible to understand why Lasso models eliminated specific coefficients?. During the modelling, many of the highly correlated features in data is being eliminated by Lasso regression. Is it ...
- 263
3
votes
1
answer
50
views
Request: Confirmation on my understanding of overfitting and regularization concepts
Overfitted models tend to have largely different (some very high, some comparatively low) coefficients/weights for different feature values. So, this means the model (when drawn as graph) will have ...
- 161
5
votes
1
answer
290
views
Visualizing effect of regularization for linear regression problem
I wanted to put together an example notebook to demonstrate how regularization makes an impact for such a simple model as a simple linear regression. When executing the below script though, I notice ...
- 299
0
votes
1
answer
42
views
1
vote
1
answer
359
views
What is the purpose of positive parameter in sklearn.linear_model.ElasticNet?
I saw this parameter in the sklearn.linear_model.ElasticNet.
What is the purpose of this? What is the possible scenario where we want to force the coefficients to ...
- 263
1
vote
1
answer
72
views
Why are we not checking the significance of the coefficients in Lasso and elastic net models
As far as I know, we don't check the coefficient significance in Lasso and elasticnet models. Is it because insignificant feature coefficients will be driven to zero in these models?. Does that mean ...
- 263
1
vote
1
answer
532
views
Elegant way to plot the L2 regularization path of logistic regression in python?
Trying to plot the L2 regularization path of logistic regression with the following code (an example of regularization path can be found in page 65 of the ML textbook Elements of Statistical Learning ...
- 41
0
votes
1
answer
62
views
Do I have to remove features with pairwise correlation even if I am doing a regularized logistic regression?
Normally we would remove features that have high pairwise correlation with another feature before performing regression. But is this step necessary if I am applying L2 regularized logistic regression (...
- 41
2
votes
1
answer
148
views
What does "regularization" actually refer to?
I am familiar with regularization, where we add a penalty in our cost function to force the model to behave a certain way. But is this a definition of regularization?
Typically we regularize to get a &...
- 168
3
votes
1
answer
717
views
Why is l1 regularization rarely used comparing to l2 regularization in Deep Learning?
l1 regularization increases sparsity, so unimportant weights are decreased closer to 0. In Deep Learning models, the input usually consists of thousands or millions of features/pixels, and the network ...
- 131
1
vote
0
answers
174
views
Adding noise after LSTM layer
I am building a Natural Language Inference neural network model that learns to identify if one sentence (hypothesis) follows from another sentence (premise). So the input to my network is 2 sentences, ...
- 21
2
votes
0
answers
83
views
How do we bring Pareto optimality into the realm of Machine Learning?
I have a multi-objective optimisation problem with a large number of objectives (more than 10) which is generally the case in most-real life problems. The use of traditional GAs such as NSGA-II or ...
3
votes
1
answer
904
views
Difference between PCA and regularisation
Currently, I am confusing about PCA and regularisation.
I wonder what is the difference between PCA and regularisation: particularly lasso (L1) regression?
Seems both of them can do the feature ...
- 133
1
vote
0
answers
32
views
Understanding model's learning curves
I'm trying to train a Lane Detection CNN called PINet on a proprietary dataset.
Below are some of the important configuration values:
Batch size: 6
Optimizer: Adam
Learning rate: High of 1e-4 and Low ...
- 11
-1
votes
1
answer
111
views
Lack of standardization in Kaggle jupyter notebooks when using lasso/ridge?
I've recently started using Kaggle, and I've noticed that for a lot of these jupyter notebooks written by others, when they use Ridge/Lasso, they don't standardize the non-categorical numerical ...
1
vote
1
answer
93
views
Why is the optimal C chosen by GridSearchCV so small?
I'm trying to use GridSearchCV to select the optimal C value in this simple SVM problem with non-separable samples. The issue I'm having is that when I run the code the optimal C is chosen to be ...
- 13
1
vote
0
answers
30
views
why use one regularisation technique over another?
why should I prefer L1 over L2, in fully-connected-layer or convolution?
why use dropout between 2 layers, when there is the option of regularising a layer(or both) with something like L1 or L2? and ...
2
votes
1
answer
53
views
Approximation of long sequence of layers by one layer
Consider the following situation : there is a deep neural network with a lot of layers, and in order to speed up the inference or for regularization purposes one would like to reduce the complexity of ...
1
vote
0
answers
603
views
Can elastic net l1 ratio be greater than 1?
I have multiple datasets that I trained with ElasticNetCV (sklearn), and I noticed that many of them selected l1_ratio = 1 as ...
- 221
1
vote
2
answers
840
views
What is the meaning of the sparsity parameter
Sparse methods such as LASSO contain a parameter $\lambda$ which is associated with the minimization of the $l_1$ norm. Higher the value of $\lambda$ ($>0$) means that more coefficients will be ...
- 541
0
votes
1
answer
72
views
Why use regularization?
In a linear model, regularization decreases the slope. Do we just assume that fitting a lin model on training data overfits by almost always creating a slope which is higher than it would be with ...
- 3
2
votes
0
answers
49
views
Entropy-regularized RL (G-learning) vs. IRL (Inverse Reinforcement Learning)
What are the differences between entropy-regularized RL (G-learning) and IRL (Inverse Reinforcement Learning)? and how are they applied to actual problems (besides stand-alone Markov decision ...
- 214
2
votes
1
answer
944
views
Is it better to use separately regularization methods for Neural Networks (L2/L1 & Dropout)
I have been exploring different regularization approaches and observed the most common to be using either Dropout Layers or L1/L2 Regularization. I have seen many debates of whether it is of interest ...
