Questions tagged [regularization]

Inclusion of additional constraints (typically a penalty for complexity) in the model fitting process. Used to prevent overfitting / enhance predictive accuracy.

Filter by
Sorted by
Tagged with
-1
votes
0answers
4 views

weight decay in ResNet50

Can someone please guide for implementing weight decay in transfer learning approach? I want to regularize the pre-trained model ResNet50, where I'm fine-tuning the model for an image classification ...
0
votes
1answer
17 views

How to reduce overfitting in a pre-trained network

I have a custom dataset with 10 classes and I am using a pre-trained resnet18 model from torch-vision. I can clearly see it's over-fitting because: the model is trained for 75 epochs with a batch size ...
1
vote
0answers
25 views

Confusion with L2 Regularization in Back-propagation

In a very simple language, this is L2 regularization $\hspace{3cm}$$Loss_R$ = $Loss_N + \sum w_i^2$ $Loss_N$ - Loss without regularization $Loss_R$ - Loss with regularization When implementing [Ref], ...
0
votes
2answers
76 views

How to handle Overfitting

I am working on machine learning classification problem with two classes (0/1). I would like to build a prediction model. The problem is that I have a small dataSet of ...
0
votes
1answer
30 views

Does stronger regularization always improve performance on testing set?

I am using the Sklearn logistic regression function to do a binary classification task on texts. I did the task using three different inputs: Bag-Of-Words, TF-IDF, Doc2vec embeddings. The question is ...
0
votes
0answers
18 views

How to interpret curve of regularization loss during CNN training?

I am fine-tuning a single shot detector (SSD) in tensorflow object detection api. I didn't freeze the backbone (mobilenet), I programmed the learning rate to go from e-3 to e-4 to e-5. In the paper ...
1
vote
3answers
40 views

Should you turn off label smoothing when validating?

As the subject says. On one hand, the answer should be yes because label smoothing is a regularization feature and how can you know if it improves performance without turning it off? On the other hand,...
0
votes
1answer
44 views

Why is my loss blowing up after adding regularization

I tried to add L2 regularization to a network class I wrote however when I train it the loss blows up even though accuracy also increases. Can someone explain where I am going wrong? (I am using the ...
1
vote
1answer
19 views

How is learning rate calculated in sklearn Lasso regression?

I was applying different regression models to Kaggle Housing dataset for advanced regression. I am planning to test out lasso, ridge and elastic net. However, none of these models have learning rate ...
5
votes
1answer
110 views

difference in l1 and l2 regularization

I have seen at different places saying that: l1 regularization penalizes weights more than l2. But the derivative of l1 norm is $\lambda$ and l2 norm is 2$\lambda$w. So l1 regularization subtracts ...
0
votes
0answers
10 views

Not regularizing bias term in gradient descent for softmax

I'm writing a gradient descent function for a multi-class classifier using softmax. I'm a bit confused about how regularization should work in the gradient function. I've specified my matrix, X, such ...
1
vote
1answer
20 views

Does regularisation make the loss noisy?

I implemented dropout and got a loss plot like this Before implementing regularisation the loss the was not noisy at all I understand why implementing dropout would increase the noise as different ...
5
votes
3answers
112 views

Problem with basic understanding of polynomial regression

I have an understanding of simple linear regression. Clear that results in a fitted line like this: However, studying polynomial regression is a bit of a challenge having some questions about the ...
2
votes
1answer
34 views

Regularization for intercept parameter

Why is the regularization parameter not applied to the intercept parameter? From what I have read about the cost functions for Linear and Logistic regression, the regularization parameter (λ) is ...
-1
votes
1answer
17 views

How does regularization help?

What is the effect of regularization on the value of parameters/weights? How does adding a regularization term in the cost function(J) and gradients help? Doesn't adding something increase the cost ...
1
vote
2answers
42 views

Should the lambda for L1 norm regularizer inversely be proportional to the number of trainable weights?

Say I want to implement Conv2D in keras and for each Conv2D layer, if I apply 20 filters of [2,3] filter on an input with depth of 10, then there will be 20*(2*3*10+1) = 1220 trainable weights. the ...
0
votes
0answers
11 views

Regularization vs Batch Normalization in CNN

I read few articles which claims that for CNN, Batch normalization will give better performance in terms training time and helps in reducing over fitting in comparison to regularization which is ...
1
vote
2answers
42 views

Why bias is not considering in Regularization?

Most of the Regularization (L1, L2 ) techniques focused mostly on the weight term only .But Regularization is not considering Bias.From my understanding large bias doesn’t make a neuron sensitive ...
0
votes
0answers
12 views

Understanding usage of dropout in Keras

I would like to check if my understanding of how dropout layers should be used in Keras training is correct. I am training pretty simple MLP regression models: ...
2
votes
2answers
69 views

On simple 1D dataset, LogisticRegressionCV selects terrible hyperparameters, resulting scores are nonsensical

I am trying to use LogisticRegressionCV to fit a logistic regression model to a simple 1D dataset. Very oddly, when given a choice, it seems to select a tiny C value, which forces my model to select a ...
5
votes
2answers
139 views

Understanding XG Boost Training (Multi class classification)

I have been working with XG boost for classification (multi class classification : 6 classes) I use 5 fold CV to train and validate my model. Please refer to the ...
0
votes
0answers
18 views

How to use Predefined Split for Randomized SearchCV

I'm trying to regularize my random forest regressor with RandomizedSearchCV. With RandomizedSearchCV the train and test are not ...
2
votes
2answers
35 views

Over fitting and association with regularization

Heard and read lot about regularization helps in reducing over fitting. But I'm not sure how exactly regularization works in reducing over fitting issue and whats the maths behind it? Appreciate if ...
3
votes
1answer
139 views

Does ridge regression always reduce coefficients by equal proportions?

Below is an excerpt from the book Introduction to statistical learning in R, (chapter-linear model selection and regularization) "In ridge regression, each least squares coefficient estimate is ...
1
vote
1answer
94 views

The meaning of random word dropout in NLP

I have been reading the early paper on pre-training in NLP (https://arxiv.org/abs/1511.01432) and I can't understand what random word dropout means. The authors completely ignore explaining this ...
1
vote
0answers
13 views

Lasso stricter with more data

I am currently analyzing investment strategies, and have implemented a backtest accordingly. This essentially means that I predict returns each month by using all prior historical data. Consequently, ...
0
votes
0answers
44 views

Training loss/accuracy fluctuates too much when using CutMix regularization

Recently I read about the CutMix data augmentation technique from this paper and I'm trying to implement it on CIFAR-10 dataset. Here is my implementation from the given pseudocode: ...
1
vote
2answers
73 views

How does L1 regularization make low-value features more zero than L2?

Below formulas, L1 and L2 regularization Many experts said that L1 regularization makes low-value features zero because of constant value. However, I think that L2 regularization could also make zero ...
0
votes
1answer
48 views

Does Sklean's SGDClassifier automatically standardize the training data when regularization is turned on?

Generally speaking--it is best to apply standarizaton (z-scoring the training data) prior to regularization. Does sklearn.linear_model.SGDClassifier automatically standardize the training data or not ...
0
votes
0answers
38 views

early-stopping changes final epoch in training each time

I am training a CNN built using transfer learning with a VGG16 network as pre-trained model, and in the training I am using early-stopping as regularization technique. I have run several time the ...
2
votes
2answers
183 views

How to build an overfitted network in order to increase performances

I am learning how to implement CNN, and searching on the internet I have found that a trick to design a good network is to first build it in such a way that it overfits, and then use regularization to ...
1
vote
1answer
259 views

How regularization helps to get rid of outliers?

I have heard regularization helps to get rid of outliers, how so? 'My intuition is, regularization shrinks parameter or even make it zero, and hence large value will have less effect on overall result'...
0
votes
0answers
8 views

What's intuition behind the activity regularizer? Any practical application?

I have read and understand that the activity regularizer is to operate on a neural layer output to make it smaller. However, I couldn't have any intuition for it and don't know why making output ...
1
vote
1answer
159 views

best way to regularize gradient boosting regressor?

i am testing gradient boosting regressor from sklearn for time series prediction on noisy data (currency markets). https://scikit-learn.org/stable/modules/generated/sklearn.ensemble....
1
vote
0answers
32 views

How can I regularize the output of a layer from scratch (without using Keras)?

I am trying to build a Convolutional Neural Network after reading notes from Stanford's cs231n course. I use ELU activation as activation function, and SoftMax as my classifier. Architecture is simple:...
0
votes
0answers
32 views

Should the LightGBM score match the regularization?

If I set the parameter objective to regression_l1 and set the metric to mean absolute error in ...
2
votes
2answers
63 views

If my model is overfitting the training dataset, does adding noise to training dataset help regularizing the machine learning model

I would like to know if this is a best practice or not. Can we add noise to the training data to help the model "fit less the training data"; as a result, hoping to generalize better on new unseen ...
0
votes
0answers
20 views

Using normal distribution data with high amplitude

I am using some data from a csv file to train a model which detects credit card fraud. The data set is from Kaggle and has 284,xxx samples each with 30 features. After some visualization in Python, ...
0
votes
0answers
16 views

Is this interpretation of spectral normalisation mathematically correct?

Hello everyone, this is my first post. I was thinking about the mathematical interpretation for spectral normalization in neural networks the other day, and I came up with an explanation that feels ...
0
votes
0answers
14 views

Binary Matrix Factorization: Regularizer to encourage 1-0 matrices

I have the following problem: Given a (user, access rights) binary matrix, I need to find the best (user, role) x (role, access right) binary matrices to reconstruct the original matrix. The current ...
1
vote
0answers
58 views

Do we need to divide our gradients by batch size our we will use the sum (Mini batch GSD plus L2 Regularization)?

I am implementing L2 regularization in C++ and I used mini batch GSD. Without L2, I was using sum of gradients during back propagation and I was not dividing my cost function by batch size. I was ...
0
votes
0answers
11 views

Regularizing Neural Network for deterministic function approximation

I'm training a neural network to learn a specific pricing function, which is entirely deterministic (i.e. same inputs always produce same outputs). The training occurs with 80 million data points from ...
0
votes
0answers
14 views

Does ridge regression reduce the coefficients of the variables all the way to zero at very high penalty

I have read in one article that ridge regression doesn't reduce the coefficients of the features to zero where as in some other article I read that it can reduce the coefficients to zero when a very ...
2
votes
3answers
556 views

Can ridge regression be used for feature selection?

I'm trying to figure out whether using Ridge Regression for regularization can be used to cause a more sparse hypothesis however to me it seems like ridge will never actually bring any coefficients to ...
2
votes
1answer
88 views

Quadratic approximation of L1 regularized cost function

I'm reading the Deep Learning book of Goodfellow, but I fail to see why minimization of (7.22) gives (7.23). I tried to compute the gradient w.r.t. the $w_{i}$ and set this to zero, but it doesn't ...
1
vote
0answers
90 views

High Variance on CNN

I'm using a shallow CNN for my current project [this one]. I have a training dataset consisting of 1000 samples and a test dataset of 400 samples. I'm using the test dataset to choose the best ...
2
votes
1answer
38 views

Why do we determine the values of λ in regularization as ln λ, such as ln λ=-18 instead of for example λ=0.3?

I'm studying Pattern Recognition and Machine Learning by Christopher Bishop. What I realized is, he defines values of λ as ln λ. For example: We see that, for a value of lnλ = −18, the over-fitting ...
1
vote
1answer
129 views

Why do we divide the regularization term by the number of examples in regularized logistic regression?

So this is the formula for the regularized logistic regression cost function: $x^{(i)}$ - the $i$'th training example $\theta_j$ - the parameter of the $j$'th feature $m$ - the number of training ...
4
votes
2answers
266 views

Why use regularization instead of decreasing the model

Regularization is used to decrease the capacity of a machine learning model to avoid overfitting. Why don't we just use a model with less capacity (e.g. decrease the number of layers). This would also ...
9
votes
2answers
4k views

L1 & L2 Regularization in Light GBM

This question pertains to L1 & L2 regularization parameters in Light GBM. As per official documentation: reg_alpha (float, optional (default=0.)) – L1 ...