Questions tagged [regularization]
Inclusion of additional constraints (typically a penalty for complexity) in the model fitting process. Used to prevent overfitting / enhance predictive accuracy.
170
questions
0
votes
1
answer
124
views
What may cause the CNN layer weight regularizer to reduce the model accuracy
What may cause the accuracy reduction when using the tf.keras regularizer at layers in CNN in the symptom? The example is simple but it happens with more complex CNN causing no improvement during the ...
- 585
0
votes
0
answers
27
views
Using Embedding For Regularization
Is using embeddings for regularization a valid practice?
My reasoning for that is that encoding training/tests datasets into smaller vectors would allow a smaller network with fewer parameters and ...
0
votes
0
answers
15
views
Making a NN closer to a linear regression?
It is possible to 'initialise' a gradient boosted model with a simpler model, such as linear regression, by manually setting the initial score. This seems to help reduce the discrepencies between the ...
- 2,576
0
votes
0
answers
11
views
Why does weight decay produce regularisation in Deep Neural Network?
Weight decay penalizes the model to have smaller weights but how does this help in regularisation?
Any intuition on smaller weights => simpler model?
0
votes
0
answers
48
views
Need feedback on idea for new regularization term
I've been working on creating a regularization term that ensures that correlated attributes are given similar weights in a linear model. This helps to avoid some of the inconsistency in the weights of ...
- 3
0
votes
1
answer
62
views
Linear Model With Highly Correlated Attributes Producing Inconsistent Weights
I know that having correlated attributes violates the linear model assumption of independent attributes, and I'm not interested in creating a more sophisticated model to tease apart the dependent ...
- 3
0
votes
0
answers
40
views
Am I using a network that is too simple for the dataset/task?
I am training an RNN to classify some high-frequency financial data. A very good performance on this data would be an accuracy of >52% or so.
I have around 650K training examples and 150K dev set ...
- 103
0
votes
0
answers
82
views
Weight decay used by Adam optimizer for neural network caused NaN validation loss
I've built a model with BCE loss for CTR prediction in which the major part is a transformer encoder. I've used 0.1 for dropout probability. When using 0 weight decay for Adam the training and ...
- 392
0
votes
0
answers
55
views
XGBoost Architecture Diagram required
Good Day!
My topic is general and theory related, about XGBoost working. I am searching XGBoost Architecture Diagram. I know it works on principles of Decision Trees, Bagging, Random Forest, Boosting, ...
- 1
0
votes
0
answers
10
views
Do we need to apply "multi-testing" corrections for the p-values in a regularized model?
Say we are fitting a penalized model, such as a linear regression with lasso regularization.
We expect to obtain a model with the most significant covariables.
The method starts with many covariables ...
- 185
0
votes
0
answers
16
views
Standardization of LASSO Regression
currently I am using the lasso regression to identify a energy function. So there is an input which is lets say x and I am creating a library of nonlinear functions of x. Those functions should ...
1
vote
0
answers
30
views
What is the l2-norm of a scalar
What is the meaning of the l2-norm when dealing with scalar values? I'm assuming it would be the same thing as taking the absolute value.
For context: I am trying to implement the clustering method ...
0
votes
1
answer
83
views
regularized LLS, trying to compute by hand the optimal weights yields wrong results
given the following dataset $S = \{(0,1),(1,1),(1,2)\}$ and the regularized problem
$$\sum_{i=1}^3 (y_i - w_1 x_i - w_0)^2 + \lambda w_1^2 \quad \lambda = 1 $$ i was tasked with finding the optimal $...
- 11
1
vote
1
answer
80
views
How can I implement some data constraints on a neural network?
I want to implement a NN model to predict the cost of my car trips. My dataset is something as follows (this is just a small sample):
...
- 123
0
votes
0
answers
121
views
Regularization in CNN change training accuracy but not validation accuracy
I have a question regarding regularization in convolutional neural networks.
So I'm building a CNN for image classification and I've come across something I don't understand.
Without using any ...
0
votes
0
answers
23
views
Data augmentation layer based on physical model for time series data
I am quite new to the Keras API, so forgive me if I use incorrect terminology and for my lack of knowledge about the API.
This is for a mathematical (wave modelling) research project and I am quite ...
- 101
0
votes
0
answers
11
views
AttributeError in pytorch
I want use https://github.com/mmbejani/TikhonovRegularizationTerm. This library includes the implementation of regularization tikhonov terms that were published from 2010 until now (2020).
I am trying ...
0
votes
1
answer
38
views
How to run a BE or FS Stepwise Regression on each dataset in a file folder full of datasets using lapply or map (without a loop)
All of the code in this question can be found in my GitHub Repository for this research project on Estimated Exhaustive Regression. Specifically, in the "Both BE & FS script" and "...
- 157
1
vote
1
answer
470
views
Hyperparameter Tuning vs Regularization
While designing the architecture of a Neural Network, should I consider adding regularization (like Dropout, L1/L2, etc.) even after optimizing the problem using Hyperparameter Tuning? What should be ...
- 77
1
vote
1
answer
33
views
Test-set error in cross-validation with regularization
I am performing regularization (Ridge regression) using cross validation. I understand that for a certain value of the regularization parameter $\lambda$, we first fit on the training-set and then we ...
- 13
1
vote
1
answer
50
views
Why does feature selection matter if your model has L1 regularization?
I've been tinkering around with boosted trees, and I saw that for common libraries there is a parameter you can set to determine L1 regularization. I doubled my original feature set to around 130 ...
- 11
0
votes
1
answer
246
views
Do Linear Regression and Logistic Regression models from sklearn include regularization?
I'm learning Data Science by enrolling on different courses, and I've recently learnt something that seems very interesting to apply when doing linear or logistic regression models: regularization.
In ...
- 25
0
votes
1
answer
38
views
Gradient tree boosting additive training
In the XGBoost documentation, they specify that the additive training is done given an objective $obj^{(t)}$ defined as
$obj^{(t)} = \sum\limits_{i=1}^n \ell(y_i, \hat{y}_i^{(t-1)}+f_t(x_i)) + \sum\...
- 137
0
votes
1
answer
351
views
Why would we add regularization loss to the gradient itself in an SVM?
I'm doing CS 231n on my own. I'm looking at this solution to a question that implements a SVM.
Relevant code:
...
- 125
1
vote
0
answers
14
views
State-of-the-art techniques for regularizing Neural Networks?
For regularizing neural networks, I'm familiar with drop-out and l2/l1 regularization, which were the biggest players in the late 2010's.
Have any significant/strong competitors risen up since then?
- 111
1
vote
1
answer
51
views
Regularizing the intercept
I am reading The Elements of Statistical Learning and regarding regularized logistic regression it says:
"As with the lasso, we typically do not penalize the intercept term"
and I am ...
- 13
0
votes
1
answer
209
views
Regularization and loss function
I am currently trying to get a better understanding of regularization as a concept. This leads me to the following question:
Will regularization change when we change the loss function?
Is it correct ...
- 125
1
vote
1
answer
110
views
Is it possible to explain why Lasso models eliminated certain coefficient?
Is it possible to understand why Lasso models eliminated specific coefficients?. During the modelling, many of the highly correlated features in data is being eliminated by Lasso regression. Is it ...
- 233
3
votes
1
answer
43
views
Request: Confirmation on my understanding of overfitting and regularization concepts
Overfitted models tend to have largely different (some very high, some comparatively low) coefficients/weights for different feature values. So, this means the model (when drawn as graph) will have ...
- 151
5
votes
1
answer
264
views
Visualizing effect of regularization for linear regression problem
I wanted to put together an example notebook to demonstrate how regularization makes an impact for such a simple model as a simple linear regression. When executing the below script though, I notice ...
- 289
0
votes
1
answer
42
views
1
vote
1
answer
242
views
What is the purpose of positive parameter in sklearn.linear_model.ElasticNet?
I saw this parameter in the sklearn.linear_model.ElasticNet.
What is the purpose of this? What is the possible scenario where we want to force the coefficients to ...
- 233
1
vote
1
answer
39
views
Why are we not checking the significance of the coefficients in Lasso and elastic net models
As far as I know, we don't check the coefficient significance in Lasso and elasticnet models. Is it because insignificant feature coefficients will be driven to zero in these models?. Does that mean ...
- 233
1
vote
1
answer
431
views
Elegant way to plot the L2 regularization path of logistic regression in python?
Trying to plot the L2 regularization path of logistic regression with the following code (an example of regularization path can be found in page 65 of the ML textbook Elements of Statistical Learning ...
- 41
0
votes
1
answer
51
views
Do I have to remove features with pairwise correlation even if I am doing a regularized logistic regression?
Normally we would remove features that have high pairwise correlation with another feature before performing regression. But is this step necessary if I am applying L2 regularized logistic regression (...
- 41
2
votes
1
answer
121
views
What does "regularization" actually refer to?
I am familiar with regularization, where we add a penalty in our cost function to force the model to behave a certain way. But is this a definition of regularization?
Typically we regularize to get a &...
- 168
2
votes
1
answer
491
views
Why is l1 regularization rarely used comparing to l2 regularization in Deep Learning?
l1 regularization increases sparsity, so unimportant weights are decreased closer to 0. In Deep Learning models, the input usually consists of thousands or millions of features/pixels, and the network ...
- 121
1
vote
0
answers
141
views
Adding noise after LSTM layer
I am building a Natural Language Inference neural network model that learns to identify if one sentence (hypothesis) follows from another sentence (premise). So the input to my network is 2 sentences, ...
- 21
2
votes
0
answers
64
views
How do we bring Pareto optimality into the realm of Machine Learning?
I have a multi-objective optimisation problem with a large number of objectives (more than 10) which is generally the case in most-real life problems. The use of traditional GAs such as NSGA-II or ...
3
votes
1
answer
674
views
Difference between PCA and regularisation
Currently, I am confusing about PCA and regularisation.
I wonder what is the difference between PCA and regularisation: particularly lasso (L1) regression?
Seems both of them can do the feature ...
- 133
1
vote
0
answers
30
views
Understanding model's learning curves
I'm trying to train a Lane Detection CNN called PINet on a proprietary dataset.
Below are some of the important configuration values:
Batch size: 6
Optimizer: Adam
Learning rate: High of 1e-4 and Low ...
- 11
-1
votes
1
answer
106
views
Lack of standardization in Kaggle jupyter notebooks when using lasso/ridge?
I've recently started using Kaggle, and I've noticed that for a lot of these jupyter notebooks written by others, when they use Ridge/Lasso, they don't standardize the non-categorical numerical ...
1
vote
1
answer
76
views
Why is the optimal C chosen by GridSearchCV so small?
I'm trying to use GridSearchCV to select the optimal C value in this simple SVM problem with non-separable samples. The issue I'm having is that when I run the code the optimal C is chosen to be ...
- 13
1
vote
0
answers
24
views
why use one regularisation technique over another?
why should I prefer L1 over L2, in fully-connected-layer or convolution?
why use dropout between 2 layers, when there is the option of regularising a layer(or both) with something like L1 or L2? and ...
2
votes
1
answer
50
views
Approximation of long sequence of layers by one layer
Consider the following situation : there is a deep neural network with a lot of layers, and in order to speed up the inference or for regularization purposes one would like to reduce the complexity of ...
1
vote
0
answers
568
views
Can elastic net l1 ratio be greater than 1?
I have multiple datasets that I trained with ElasticNetCV (sklearn), and I noticed that many of them selected l1_ratio = 1 as ...
- 221
1
vote
2
answers
645
views
What is the meaning of the sparsity parameter
Sparse methods such as LASSO contain a parameter $\lambda$ which is associated with the minimization of the $l_1$ norm. Higher the value of $\lambda$ ($>0$) means that more coefficients will be ...
- 521
0
votes
1
answer
70
views
Why use regularization?
In a linear model, regularization decreases the slope. Do we just assume that fitting a lin model on training data overfits by almost always creating a slope which is higher than it would be with ...
- 3
2
votes
0
answers
42
views
Entropy-regularized RL (G-learning) vs. IRL (Inverse Reinforcement Learning)
What are the differences between entropy-regularized RL (G-learning) and IRL (Inverse Reinforcement Learning)? and how are they applied to actual problems (besides stand-alone Markov decision ...
- 214
2
votes
1
answer
714
views
Is it better to use separately regularization methods for Neural Networks (L2/L1 & Dropout)
I have been exploring different regularization approaches and observed the most common to be using either Dropout Layers or L1/L2 Regularization. I have seen many debates of whether it is of interest ...
