Skip to main content
Share Your Experience: Take the 2024 Developer Survey

Questions tagged [regularization]

Inclusion of additional constraints (typically a penalty for complexity) in the model fitting process. Used to prevent overfitting / enhance predictive accuracy.

Filter by
Sorted by
Tagged with
0 votes
0 answers
12 views

Implementing L2 Regularization in pure NumPy

I was implementing L2 Regularization in pure NumPy as an exercise to figure out the inner workings of the mathematics of an ML model. I'm unsure if it's done well, I don't really have a reference to ...
vxnuaj's user avatar
  • 11
0 votes
0 answers
23 views

Adaptive Lasso Coefficient Weights

I'm trying to understand how the Adaptive part of Adaptive Lasso works. I understand that theoretically, the weights for zero coefficients are inflated to infinity. But can someone explain this ...
user162172's user avatar
0 votes
0 answers
21 views

Question about the limitations of regularization

I am training a neural network which is overfitting. Even when I increase the number of parameters, the test lost plateaus while the training loss keeps decreasing. Can regularization (like an L1 or ...
vermillion flycatcher's user avatar
2 votes
1 answer
53 views

What does it mean if a neural networks starts overfitting more after applying regularisation techniques

Background I am building a CNN to categorize cytometric cell data into healthy and diseased groups. The architecture looks as follows: 3 Convolutional layers followed by average pooling followed by 3 ...
Viktor VN's user avatar
  • 121
0 votes
1 answer
149 views

What may cause the CNN layer weight regularizer to reduce the model accuracy

What may cause the accuracy reduction when using the tf.keras regularizer at layers in CNN in the symptom? The example is simple but it happens with more complex CNN causing no improvement during the ...
mon's user avatar
  • 721
0 votes
0 answers
28 views

Using Embedding For Regularization

Is using embeddings for regularization a valid practice? My reasoning for that is that encoding training/tests datasets into smaller vectors would allow a smaller network with fewer parameters and ...
Adenilson Arcanjo's user avatar
0 votes
0 answers
17 views

Making a NN closer to a linear regression?

It is possible to 'initialise' a gradient boosted model with a simpler model, such as linear regression, by manually setting the initial score. This seems to help reduce the discrepencies between the ...
Lucas Morin's user avatar
  • 2,244
0 votes
0 answers
17 views

Why does weight decay produce regularisation in Deep Neural Network?

Weight decay penalizes the model to have smaller weights but how does this help in regularisation? Any intuition on smaller weights => simpler model?
Sushil Khadka's user avatar
0 votes
0 answers
48 views

Need feedback on idea for new regularization term

I've been working on creating a regularization term that ensures that correlated attributes are given similar weights in a linear model. This helps to avoid some of the inconsistency in the weights of ...
Brett L's user avatar
0 votes
1 answer
81 views

Linear Model With Highly Correlated Attributes Producing Inconsistent Weights

I know that having correlated attributes violates the linear model assumption of independent attributes, and I'm not interested in creating a more sophisticated model to tease apart the dependent ...
Brett L's user avatar
0 votes
0 answers
41 views

Am I using a network that is too simple for the dataset/task?

I am training an RNN to classify some high-frequency financial data. A very good performance on this data would be an accuracy of >52% or so. I have around 650K training examples and 150K dev set ...
BYZZav's user avatar
  • 103
0 votes
0 answers
177 views

Weight decay used by Adam optimizer for neural network caused NaN validation loss

I've built a model with BCE loss for CTR prediction in which the major part is a transformer encoder. I've used 0.1 for dropout probability. When using 0 weight decay for Adam the training and ...
CyberPlayerOne's user avatar
0 votes
0 answers
70 views

XGBoost Architecture Diagram required

Good Day! My topic is general and theory related, about XGBoost working. I am searching XGBoost Architecture Diagram. I know it works on principles of Decision Trees, Bagging, Random Forest, Boosting, ...
P_Z's user avatar
  • 101
0 votes
0 answers
12 views

Do we need to apply "multi-testing" corrections for the p-values in a regularized model?

Say we are fitting a penalized model, such as a linear regression with lasso regularization. We expect to obtain a model with the most significant covariables. The method starts with many covariables ...
skan's user avatar
  • 185
0 votes
0 answers
25 views

Standardization of LASSO Regression

currently I am using the lasso regression to identify a energy function. So there is an input which is lets say x and I am creating a library of nonlinear functions of x. Those functions should ...
user150587's user avatar
1 vote
0 answers
39 views

What is the l2-norm of a scalar

What is the meaning of the l2-norm when dealing with scalar values? I'm assuming it would be the same thing as taking the absolute value. For context: I am trying to implement the clustering method ...
Droidenkiller's user avatar
0 votes
1 answer
85 views

regularized LLS, trying to compute by hand the optimal weights yields wrong results

given the following dataset $S = \{(0,1),(1,1),(1,2)\}$ and the regularized problem $$\sum_{i=1}^3 (y_i - w_1 x_i - w_0)^2 + \lambda w_1^2 \quad \lambda = 1 $$ i was tasked with finding the optimal $...
kal_elk122's user avatar
1 vote
1 answer
112 views

How can I implement some data constraints on a neural network?

I want to implement a NN model to predict the cost of my car trips. My dataset is something as follows (this is just a small sample): ...
Dani's user avatar
  • 123
0 votes
0 answers
23 views

Data augmentation layer based on physical model for time series data

I am quite new to the Keras API, so forgive me if I use incorrect terminology and for my lack of knowledge about the API. This is for a mathematical (wave modelling) research project and I am quite ...
LightninBolt74's user avatar
0 votes
1 answer
38 views

How to run a BE or FS Stepwise Regression on each dataset in a file folder full of datasets using lapply or map (without a loop)

All of the code in this question can be found in my GitHub Repository for this research project on Estimated Exhaustive Regression. Specifically, in the "Both BE & FS script" and "...
Marlen's user avatar
  • 167
1 vote
1 answer
799 views

Hyperparameter Tuning vs Regularization

While designing the architecture of a Neural Network, should I consider adding regularization (like Dropout, L1/L2, etc.) even after optimizing the problem using Hyperparameter Tuning? What should be ...
Harsh Khare's user avatar
1 vote
1 answer
36 views

Test-set error in cross-validation with regularization

I am performing regularization (Ridge regression) using cross validation. I understand that for a certain value of the regularization parameter $\lambda$, we first fit on the training-set and then we ...
dimitris's user avatar
1 vote
1 answer
99 views

Why does feature selection matter if your model has L1 regularization?

I've been tinkering around with boosted trees, and I saw that for common libraries there is a parameter you can set to determine L1 regularization. I doubled my original feature set to around 130 ...
ron burgundy's user avatar
0 votes
1 answer
335 views

Do Linear Regression and Logistic Regression models from sklearn include regularization?

I'm learning Data Science by enrolling on different courses, and I've recently learnt something that seems very interesting to apply when doing linear or logistic regression models: regularization. In ...
Álvaro V.'s user avatar
0 votes
1 answer
45 views

Gradient tree boosting additive training

In the XGBoost documentation, they specify that the additive training is done given an objective $obj^{(t)}$ defined as $obj^{(t)} = \sum\limits_{i=1}^n \ell(y_i, \hat{y}_i^{(t-1)}+f_t(x_i)) + \sum\...
Hadar's user avatar
  • 157
0 votes
1 answer
510 views

Why would we add regularization loss to the gradient itself in an SVM?

I'm doing CS 231n on my own. I'm looking at this solution to a question that implements a SVM. Relevant code: ...
Foobar's user avatar
  • 125
1 vote
0 answers
14 views

State-of-the-art techniques for regularizing Neural Networks?

For regularizing neural networks, I'm familiar with drop-out and l2/l1 regularization, which were the biggest players in the late 2010's. Have any significant/strong competitors risen up since then?
chausies's user avatar
  • 111
1 vote
1 answer
78 views

Regularizing the intercept

I am reading The Elements of Statistical Learning and regarding regularized logistic regression it says: "As with the lasso, we typically do not penalize the intercept term" and I am ...
ChuckNoise's user avatar
0 votes
1 answer
240 views

Regularization and loss function

I am currently trying to get a better understanding of regularization as a concept. This leads me to the following question: Will regularization change when we change the loss function? Is it correct ...
Piskator's user avatar
  • 135
1 vote
1 answer
155 views

Is it possible to explain why Lasso models eliminated certain coefficient?

Is it possible to understand why Lasso models eliminated specific coefficients?. During the modelling, many of the highly correlated features in data is being eliminated by Lasso regression. Is it ...
NAS_2339's user avatar
  • 243
3 votes
1 answer
45 views

Request: Confirmation on my understanding of overfitting and regularization concepts

Overfitted models tend to have largely different (some very high, some comparatively low) coefficients/weights for different feature values. So, this means the model (when drawn as graph) will have ...
Curious's user avatar
  • 161
5 votes
1 answer
282 views

Visualizing effect of regularization for linear regression problem

I wanted to put together an example notebook to demonstrate how regularization makes an impact for such a simple model as a simple linear regression. When executing the below script though, I notice ...
lazarea's user avatar
  • 299
0 votes
1 answer
42 views

What is the problem that causes overfitting in the code?

** ...
Esra Ibra's user avatar
1 vote
1 answer
310 views

What is the purpose of positive parameter in sklearn.linear_model.ElasticNet?

I saw this parameter in the sklearn.linear_model.ElasticNet. What is the purpose of this? What is the possible scenario where we want to force the coefficients to ...
NAS_2339's user avatar
  • 243
1 vote
1 answer
57 views

Why are we not checking the significance of the coefficients in Lasso and elastic net models

As far as I know, we don't check the coefficient significance in Lasso and elasticnet models. Is it because insignificant feature coefficients will be driven to zero in these models?. Does that mean ...
NAS_2339's user avatar
  • 243
1 vote
1 answer
503 views

Elegant way to plot the L2 regularization path of logistic regression in python?

Trying to plot the L2 regularization path of logistic regression with the following code (an example of regularization path can be found in page 65 of the ML textbook Elements of Statistical Learning ...
lostwanderer's user avatar
0 votes
1 answer
57 views

Do I have to remove features with pairwise correlation even if I am doing a regularized logistic regression?

Normally we would remove features that have high pairwise correlation with another feature before performing regression. But is this step necessary if I am applying L2 regularized logistic regression (...
lostwanderer's user avatar
2 votes
1 answer
140 views

What does "regularization" actually refer to?

I am familiar with regularization, where we add a penalty in our cost function to force the model to behave a certain way. But is this a definition of regularization? Typically we regularize to get a &...
jtb's user avatar
  • 168
3 votes
1 answer
631 views

Why is l1 regularization rarely used comparing to l2 regularization in Deep Learning?

l1 regularization increases sparsity, so unimportant weights are decreased closer to 0. In Deep Learning models, the input usually consists of thousands or millions of features/pixels, and the network ...
seermer's user avatar
  • 131
1 vote
0 answers
165 views

Adding noise after LSTM layer

I am building a Natural Language Inference neural network model that learns to identify if one sentence (hypothesis) follows from another sentence (premise). So the input to my network is 2 sentences, ...
lmtr339's user avatar
  • 21
2 votes
0 answers
73 views

How do we bring Pareto optimality into the realm of Machine Learning?

I have a multi-objective optimisation problem with a large number of objectives (more than 10) which is generally the case in most-real life problems. The use of traditional GAs such as NSGA-II or ...
thatbangaloreanguy's user avatar
3 votes
1 answer
803 views

Difference between PCA and regularisation

Currently, I am confusing about PCA and regularisation. I wonder what is the difference between PCA and regularisation: particularly lasso (L1) regression? Seems both of them can do the feature ...
Crazy's user avatar
  • 133
1 vote
0 answers
32 views

Understanding model's learning curves

I'm trying to train a Lane Detection CNN called PINet on a proprietary dataset. Below are some of the important configuration values: Batch size: 6 Optimizer: Adam Learning rate: High of 1e-4 and Low ...
Vishal's user avatar
  • 11
-1 votes
1 answer
110 views

Lack of standardization in Kaggle jupyter notebooks when using lasso/ridge?

I've recently started using Kaggle, and I've noticed that for a lot of these jupyter notebooks written by others, when they use Ridge/Lasso, they don't standardize the non-categorical numerical ...
student010101's user avatar
1 vote
1 answer
86 views

Why is the optimal C chosen by GridSearchCV so small?

I'm trying to use GridSearchCV to select the optimal C value in this simple SVM problem with non-separable samples. The issue I'm having is that when I run the code the optimal C is chosen to be ...
Nitram's user avatar
  • 13
1 vote
0 answers
29 views

why use one regularisation technique over another?

why should I prefer L1 over L2, in fully-connected-layer or convolution? why use dropout between 2 layers, when there is the option of regularising a layer(or both) with something like L1 or L2? and ...
Naveen Reddy Marthala's user avatar
2 votes
1 answer
53 views

Approximation of long sequence of layers by one layer

Consider the following situation : there is a deep neural network with a lot of layers, and in order to speed up the inference or for regularization purposes one would like to reduce the complexity of ...
spiridon_the_sun_rotator's user avatar
1 vote
0 answers
587 views

Can elastic net l1 ratio be greater than 1?

I have multiple datasets that I trained with ElasticNetCV (sklearn), and I noticed that many of them selected l1_ratio = 1 as ...
Oren Matar's user avatar
1 vote
2 answers
760 views

What is the meaning of the sparsity parameter

Sparse methods such as LASSO contain a parameter $\lambda$ which is associated with the minimization of the $l_1$ norm. Higher the value of $\lambda$ ($>0$) means that more coefficients will be ...
Sm1's user avatar
  • 541
0 votes
1 answer
70 views

Why use regularization?

In a linear model, regularization decreases the slope. Do we just assume that fitting a lin model on training data overfits by almost always creating a slope which is higher than it would be with ...
AUser240's user avatar