Questions tagged [regularization]

Inclusion of additional constraints (typically a penalty for complexity) in the model fitting process. Used to prevent overfitting / enhance predictive accuracy.

Filter by
Sorted by
Tagged with
2
votes
1answer
16 views

Approximation of long sequence of layers by one layer

Consider the following situation : there is a deep neural network with a lot of layers, and in order to speed up the inference or for regularization purposes one would like to reduce the complexity of ...
0
votes
0answers
16 views

Regression for prediction: is there any benefit of regularization?

What is the benefit of having flatter regression line in linear regression? Is there a proven benefit for prediction? Are there experiments that show regression with regularization performs better on ...
1
vote
0answers
23 views

Can elastic net l1 ratio be greater than 1?

I have multiple datasets that I trained with ElasticNetCV (sklearn), and I noticed that many of them selected l1_ratio = 1 as ...
1
vote
2answers
79 views

What is the meaning of the sparsity parameter

Sparse methods such as LASSO contains a parameter $\lambda$ which is associated with the minimization of the $l_1$ norm. Higher the value of $\lambda$ ($>0$) it means that more coefficients will be ...
0
votes
1answer
36 views

Why use regularization?

In a linear model, regularization decreases the slope. Do we just assume that fitting a lin model on training data overfits by almost always creating a slope which is higher than it would be with ...
2
votes
0answers
18 views

Entropy-regularized RL (G-learning) vs. IRL (Inverse Reinforcement Learning)

What are the differences between entropy-regularized RL (G-learning) and IRL (Inverse Reinforcement Learning)? and how are they applied to actual problems (besides stand-alone Markov decision ...
2
votes
1answer
86 views

Is it better to use separately regularization methods for Neural Networks (L2/L1 & Dropout)

I have been exploring different regularization approaches and observed the most common to be using either Dropout Layers or L1/L2 Regularization. I have seen many debates of whether it is of interest ...
1
vote
0answers
13 views

What approach would you use to determine the effects of an event when you don't know the effects?

Let's say you have a problem where in time 0 you have an event and then you want to figure out what effects are caused by the event in time 1. Unfortunately, you do not know what the effects are, but ...
3
votes
1answer
44 views

Relatively high regularization parameters for XGBoost model only way to prevent overfitting

I am modeling a continuous regression/forecasting problem for very right-skewed data. I've been using ElasticNet and Huber regression with quite a bit of success, and have recently moved into using ...
3
votes
2answers
72 views

When I add regularization like L1,L2 , do I need more epochs to properly train my model?

When I add regularization techniques in my model like L1 or L2 do i need more epochs to properly converge my model. ...
0
votes
1answer
35 views

Regularization hyperparam tuning during training

I have an idea for a regularization-hyperparam selection method, which I haven't encountered before and can't find on Google, but I'm sure someone has already tried it and I'm wondering what are the ...
3
votes
1answer
28 views

What's the difference between hessian regularisation (min_child_weight) and loss regularisation (gamma)? When to use one over another?

I wonder about the difference between min_child_weight and gamma regularisation in XGBoost. From my understanding: hessian ...
1
vote
1answer
390 views

What is C in sklearn Logistic Regression?

In sklearn.linear_model.LogisticRegression, there is a parameter C according to docs Cfloat, default=1.0 Inverse of ...
1
vote
1answer
105 views

Dropping one category for regularized linear models

While reviewing the sklearn's OneHotEncoder documentation (attached below) I noticed that when applying regularization (e.g., lasso, ridge, etc.) it is not recommended to drop the first category. ...
1
vote
1answer
47 views

Why is Regularization after PCA or Factor Analysis a bad idea?

I have done Factor Analysis on my data and applied various machine learning models on it. I particularly find it giving high MSE value for Ridge and Lasso Regression compared to other models. I want ...
0
votes
1answer
145 views

Correct theoretical regularized objective function for XGB/LGBM (regression task)

I am writing an academic paper on the application of Machine Learning methods to Time Series Forecasting and I am unsure about how to write down the theoretical part about the regularized objective ...
2
votes
2answers
69 views

What is the intuition behind decreasing the slope when using regularization?

While training a logistic regression model, using regularization can help distribute weights and avoid reliance on some particular weight, making the model more robust. Eg: suppose my input vector is ...
0
votes
0answers
30 views

Sparse activity regularization for DL

I am trying to find some kind of activation regularization that will have the following effect: assuming that the layer has output shape (batch_size,4), the regularization will force it instead of ...
1
vote
0answers
26 views

Do non-parametric models always overfit without regularization?

Let's scope this to just classification. It's clear that if you fully grow out a decision tree with no regularization (e.g. max depth, pruning), it will overfit the training data and get full accuracy ...
1
vote
1answer
45 views

regularization error vs over fitting

I have a collected data from 50 unique blocks, and then merged data from 49 blocks into one data set, and saved the data from 1 block for testing purpose. I then split the merged data set from 49 ...
0
votes
0answers
88 views

weight decay in ResNet50

Can someone please guide for implementing weight decay in transfer learning approach? I want to regularize the pre-trained model ResNet50, where I'm fine-tuning the model for an image classification ...
1
vote
1answer
323 views

How to reduce overfitting in a pre-trained network

I have a custom dataset with 10 classes and I am using a pre-trained resnet18 model from torch-vision. I can clearly see it's over-fitting because: the model is trained for 75 epochs with a batch size ...
2
votes
0answers
67 views

Confusion with L2 Regularization in Back-propagation

In a very simple language, this is L2 regularization $\hspace{3cm}$$Loss_R$ = $Loss_N + \sum w_i^2$ $Loss_N$ - Loss without regularization $Loss_R$ - Loss with regularization When implementing [Ref], ...
2
votes
2answers
272 views

How to handle Overfitting

I am working on machine learning classification problem with two classes (0/1). I would like to build a prediction model. The problem is that I have a small dataSet of ...
0
votes
1answer
84 views

Does stronger regularization always improve performance on testing set?

I am using the Sklearn logistic regression function to do a binary classification task on texts. I did the task using three different inputs: Bag-Of-Words, TF-IDF, Doc2vec embeddings. The question is ...
0
votes
0answers
111 views

How to interpret curve of regularization loss during CNN training?

I am fine-tuning a single shot detector (SSD) in tensorflow object detection api. I didn't freeze the backbone (mobilenet), I programmed the learning rate to go from e-3 to e-4 to e-5. In the paper ...
2
votes
3answers
107 views

Should you turn off label smoothing when validating?

As the subject says. On one hand, the answer should be yes because label smoothing is a regularization feature and how can you know if it improves performance without turning it off? On the other hand,...
1
vote
1answer
109 views

Why is my loss blowing up after adding regularization

I tried to add L2 regularization to a network class I wrote however when I train it the loss blows up even though accuracy also increases. Can someone explain where I am going wrong? (I am using the ...
1
vote
1answer
131 views

How is learning rate calculated in sklearn Lasso regression?

I was applying different regression models to Kaggle Housing dataset for advanced regression. I am planning to test out lasso, ridge and elastic net. However, none of these models have learning rate ...
5
votes
1answer
294 views

difference in l1 and l2 regularization

I have seen at different places saying that: l1 regularization penalizes weights more than l2. But the derivative of l1 norm is $\lambda$ and l2 norm is 2$\lambda$w. So l1 regularization subtracts ...
0
votes
0answers
25 views

Not regularizing bias term in gradient descent for softmax

I'm writing a gradient descent function for a multi-class classifier using softmax. I'm a bit confused about how regularization should work in the gradient function. I've specified my matrix, X, such ...
1
vote
1answer
29 views

Does regularisation make the loss noisy?

I implemented dropout and got a loss plot like this Before implementing regularisation the loss the was not noisy at all I understand why implementing dropout would increase the noise as different ...
5
votes
3answers
141 views

Problem with basic understanding of polynomial regression

I have an understanding of simple linear regression. Clear that results in a fitted line like this: However, studying polynomial regression is a bit of a challenge having some questions about the ...
2
votes
1answer
71 views

Regularization for intercept parameter

Why is the regularization parameter not applied to the intercept parameter? From what I have read about the cost functions for Linear and Logistic regression, the regularization parameter (λ) is ...
-1
votes
1answer
23 views

How does regularization help?

What is the effect of regularization on the value of parameters/weights? How does adding a regularization term in the cost function(J) and gradients help? Doesn't adding something increase the cost ...
1
vote
2answers
91 views

Should the lambda for L1 norm regularizer inversely be proportional to the number of trainable weights?

Say I want to implement Conv2D in keras and for each Conv2D layer, if I apply 20 filters of [2,3] filter on an input with depth of 10, then there will be 20*(2*3*10+1) = 1220 trainable weights. the ...
0
votes
0answers
16 views

Regularization vs Batch Normalization in CNN

I read few articles which claims that for CNN, Batch normalization will give better performance in terms training time and helps in reducing over fitting in comparison to regularization which is ...
1
vote
2answers
58 views

Why bias is not considering in Regularization?

Most of the Regularization (L1, L2 ) techniques focused mostly on the weight term only .But Regularization is not considering Bias.From my understanding large bias doesn’t make a neuron sensitive ...
0
votes
0answers
21 views

Understanding usage of dropout in Keras

I would like to check if my understanding of how dropout layers should be used in Keras training is correct. I am training pretty simple MLP regression models: ...
3
votes
2answers
113 views

On simple 1D dataset, LogisticRegressionCV selects terrible hyperparameters, resulting scores are nonsensical

I am trying to use LogisticRegressionCV to fit a logistic regression model to a simple 1D dataset. Very oddly, when given a choice, it seems to select a tiny C value, which forces my model to select a ...
5
votes
2answers
364 views

Understanding XG Boost Training (Multi class classification)

I have been working with XG boost for classification (multi class classification : 6 classes) I use 5 fold CV to train and validate my model. Please refer to the ...
0
votes
0answers
89 views

How to use Predefined Split for Randomized SearchCV

I'm trying to regularize my random forest regressor with RandomizedSearchCV. With RandomizedSearchCV the train and test are not ...
2
votes
2answers
43 views

Over fitting and association with regularization

Heard and read lot about regularization helps in reducing over fitting. But I'm not sure how exactly regularization works in reducing over fitting issue and whats the maths behind it? Appreciate if ...
3
votes
1answer
162 views

Does ridge regression always reduce coefficients by equal proportions?

Below is an excerpt from the book Introduction to statistical learning in R, (chapter-linear model selection and regularization) "In ridge regression, each least squares coefficient estimate is ...
1
vote
1answer
288 views

The meaning of random word dropout in NLP

I have been reading the early paper on pre-training in NLP (https://arxiv.org/abs/1511.01432) and I can't understand what random word dropout means. The authors completely ignore explaining this ...
1
vote
0answers
15 views

Lasso stricter with more data

I am currently analyzing investment strategies, and have implemented a backtest accordingly. This essentially means that I predict returns each month by using all prior historical data. Consequently, ...
1
vote
2answers
143 views

How does L1 regularization make low-value features more zero than L2?

Below formulas, L1 and L2 regularization Many experts said that L1 regularization makes low-value features zero because of constant value. However, I think that L2 regularization could also make zero ...
0
votes
1answer
136 views

Does Sklean's SGDClassifier automatically standardize the training data when regularization is turned on?

Generally speaking--it is best to apply standarizaton (z-scoring the training data) prior to regularization. Does sklearn.linear_model.SGDClassifier automatically standardize the training data or not ...
0
votes
0answers
73 views

early-stopping changes final epoch in training each time

I am training a CNN built using transfer learning with a VGG16 network as pre-trained model, and in the training I am using early-stopping as regularization technique. I have run several time the ...
2
votes
2answers
191 views

How to build an overfitted network in order to increase performances

I am learning how to implement CNN, and searching on the internet I have found that a trick to design a good network is to first build it in such a way that it overfits, and then use regularization to ...