Questions tagged [regularization]

Inclusion of additional constraints (typically a penalty for complexity) in the model fitting process. Used to prevent overfitting / enhance predictive accuracy.

Filter by
Sorted by
Tagged with
0
votes
0answers
14 views

How to use Predefined Split for Randomized SearchCV

I'm trying to regularize my random forest regressor with RandomizedSearchCV. With RandomizedSearchCV the train and test are not ...
2
votes
2answers
28 views

Over fitting and association with regularization

Heard and read lot about regularization helps in reducing over fitting. But I'm not sure how exactly regularization works in reducing over fitting issue and whats the maths behind it? Appreciate if ...
3
votes
1answer
99 views

Does ridge regression always reduce coefficients by equal proportions?

Below is an excerpt from the book Introduction to statistical learning in R, (chapter-linear model selection and regularization) "In ridge regression, each least squares coefficient estimate is ...
1
vote
1answer
45 views

The meaning of random word dropout in NLP

I have been reading the early paper on pre-training in NLP (https://arxiv.org/abs/1511.01432) and I can't understand what random word dropout means. The authors completely ignore explaining this ...
1
vote
0answers
12 views

Lasso stricter with more data

I am currently analyzing investment strategies, and have implemented a backtest accordingly. This essentially means that I predict returns each month by using all prior historical data. Consequently, ...
0
votes
0answers
24 views

Training loss/accuracy fluctuates too much when using CutMix regularization

Recently I read about the CutMix data augmentation technique from this paper and I'm trying to implement it on CIFAR-10 dataset. Here is my implementation from the given pseudocode: ...
1
vote
2answers
50 views

How does L1 regularization make low-value features more zero than L2?

Below formulas, L1 and L2 regularization Many experts said that L1 regularization makes low-value features zero because of constant value. However, I think that L2 regularization could also make zero ...
0
votes
1answer
22 views

Does Sklean's SGDClassifier automatically standardize the training data when regularization is turned on?

Generally speaking--it is best to apply standarizaton (z-scoring the training data) prior to regularization. Does sklearn.linear_model.SGDClassifier automatically standardize the training data or not ...
0
votes
0answers
15 views

early-stopping changes final epoch in training each time

I am training a CNN built using transfer learning with a VGG16 network as pre-trained model, and in the training I am using early-stopping as regularization technique. I have run several time the ...
2
votes
2answers
178 views

How to build an overfitted network in order to increase performances

I am learning how to implement CNN, and searching on the internet I have found that a trick to design a good network is to first build it in such a way that it overfits, and then use regularization to ...
1
vote
1answer
89 views

How regularization helps to get rid of outliers?

I have heard regularization helps to get rid of outliers, how so? 'My intuition is, regularization shrinks parameter or even make it zero, and hence large value will have less effect on overall result'...
0
votes
0answers
8 views

What's intuition behind the activity regularizer? Any practical application?

I have read and understand that the activity regularizer is to operate on a neural layer output to make it smaller. However, I couldn't have any intuition for it and don't know why making output ...
1
vote
1answer
48 views

best way to regularize gradient boosting regressor?

i am testing gradient boosting regressor from sklearn for time series prediction on noisy data (currency markets). https://scikit-learn.org/stable/modules/generated/sklearn.ensemble....
1
vote
0answers
28 views

How can I regularize the output of a layer from scratch (without using Keras)?

I am trying to build a Convolutional Neural Network after reading notes from Stanford's cs231n course. I use ELU activation as activation function, and SoftMax as my classifier. Architecture is simple:...
0
votes
0answers
26 views

Should the LightGBM score match the regularization?

If I set the parameter objective to regression_l1 and set the metric to mean absolute error in ...
2
votes
2answers
57 views

If my model is overfitting the training dataset, does adding noise to training dataset help regularizing the machine learning model

I would like to know if this is a best practice or not. Can we add noise to the training data to help the model "fit less the training data"; as a result, hoping to generalize better on new unseen ...
0
votes
0answers
20 views

Using normal distribution data with high amplitude

I am using some data from a csv file to train a model which detects credit card fraud. The data set is from Kaggle and has 284,xxx samples each with 30 features. After some visualization in Python, ...
0
votes
0answers
14 views

Is this interpretation of spectral normalisation mathematically correct?

Hello everyone, this is my first post. I was thinking about the mathematical interpretation for spectral normalization in neural networks the other day, and I came up with an explanation that feels ...
0
votes
0answers
14 views

Binary Matrix Factorization: Regularizer to encourage 1-0 matrices

I have the following problem: Given a (user, access rights) binary matrix, I need to find the best (user, role) x (role, access right) binary matrices to reconstruct the original matrix. The current ...
1
vote
0answers
30 views

Do we need to divide our gradients by batch size our we will use the sum (Mini batch GSD plus L2 Regularization)?

I am implementing L2 regularization in C++ and I used mini batch GSD. Without L2, I was using sum of gradients during back propagation and I was not dividing my cost function by batch size. I was ...
0
votes
0answers
11 views

Regularizing Neural Network for deterministic function approximation

I'm training a neural network to learn a specific pricing function, which is entirely deterministic (i.e. same inputs always produce same outputs). The training occurs with 80 million data points from ...
0
votes
0answers
11 views

Does ridge regression reduce the coefficients of the variables all the way to zero at very high penalty

I have read in one article that ridge regression doesn't reduce the coefficients of the features to zero where as in some other article I read that it can reduce the coefficients to zero when a very ...
2
votes
3answers
134 views

Can ridge regression be used for feature selection?

I'm trying to figure out whether using Ridge Regression for regularization can be used to cause a more sparse hypothesis however to me it seems like ridge will never actually bring any coefficients to ...
2
votes
1answer
72 views

Quadratic approximation of L1 regularized cost function

I'm reading the Deep Learning book of Goodfellow, but I fail to see why minimization of (7.22) gives (7.23). I tried to compute the gradient w.r.t. the $w_{i}$ and set this to zero, but it doesn't ...
1
vote
0answers
57 views

High Variance on CNN

I'm using a shallow CNN for my current project [this one]. I have a training dataset consisting of 1000 samples and a test dataset of 400 samples. I'm using the test dataset to choose the best ...
2
votes
1answer
34 views

Why do we determine the values of λ in regularization as ln λ, such as ln λ=-18 instead of for example λ=0.3?

I'm studying Pattern Recognition and Machine Learning by Christopher Bishop. What I realized is, he defines values of λ as ln λ. For example: We see that, for a value of lnλ = −18, the over-fitting ...
1
vote
1answer
74 views

Why do we divide the regularization term by the number of examples in regularized logistic regression?

So this is the formula for the regularized logistic regression cost function: $x^{(i)}$ - the $i$'th training example $\theta_j$ - the parameter of the $j$'th feature $m$ - the number of training ...
4
votes
2answers
217 views

Why use regularization instead of decreasing the model

Regularization is used to decrease the capacity of a machine learning model to avoid overfitting. Why don't we just use a model with less capacity (e.g. decrease the number of layers). This would also ...
5
votes
2answers
2k views

L1 & L2 Regularization in Light GBM

This question pertains to L1 & L2 regularization parameters in Light GBM. As per official documentation: reg_alpha (float, optional (default=0.)) – L1 ...
2
votes
1answer
532 views

Light GBM Regressor, L1 & L2 Regularization and Feature Importances

I want to know how L1 & L2 regularization works in Light GBM and how to interpret the feature importances. Scenario is: I used LGBM Regressor with RandomizedSearchCV (cv=3, iterations=50) on a ...
4
votes
2answers
100 views

Why does Lasso behave “erratically” when the number of features is greater than the number of training instances?

From the book "Hands-on Machine Learning with Scikit-Learn and TensorFlow 2nd edition" chapter 4: In general, Elastic Net is preferred over Lasso since Lasso may behave erratically when the ...
1
vote
1answer
60 views

Improving Accuracy of the Deep Learning Model

In my current project, I have only 647 rows (500 for training and 147 for testing) and I have applied the Keras Sequential model using the following code: ...
1
vote
0answers
28 views

Version of Perceptron

If we change the $ywx<0$ condition (for performing update) to $ywx<1$ like in SVM (but without adding regularization to maximize the margin), is there any difference from the basic perceptron (...
1
vote
1answer
83 views

Why non-differentiable regularization lead to setting coefficients to 0?

The L2 regularization lead to minimize the values in the vector parameter. The L1 regularization lead to setting some coefficients to 0 in the vector parameter. More generally, I've seen that non-...
0
votes
0answers
26 views

Regularization loss - Why is that important to get unique weights solution

There's something that is bothering me in Regularization. There is one bug with the loss function when I just use for e.g. a multi-class SVM loss. Suppose that we have a dataset and a set of ...
1
vote
2answers
203 views

Difference between LASSO penalty in neural network and just LASSO regression

I wonder whether those two have any significant differences. I think in neural network, the lasso penalty put on the loss function makes the model simpler and introduces more sparsity by ...
3
votes
2answers
107 views

SVM behavior when regularization parameter equals 0

I read on this Wikipedia page the following about soft-margin SVM: "The parameter $λ$ determines the trade-off between increasing the margin size and ensuring that the $x_i$ lie on the correct ...
2
votes
1answer
106 views

Multiclass classification with high number of classes, high number of features and small sample size

I am working on a biology related dataset with over 300K features, and I only have about 5K samples. I want my model to classify many classes. For this problem in particular the class is age. Each age ...
1
vote
1answer
46 views

correct ML approach

I wanted to get your thoughts on a problem I have been facing. I have daily level product sales information (about 4 years). The sales are affected by the typical factors such as seasonality, day of ...
0
votes
0answers
16 views

what is the relationship between Penalties and Regularization?

i am learning the DeepLearningBook, in which chapter7 talks a lot about Penalties and Regularization. i would like to figure out the relationship between Penalties and Regularization. it seems that ...
0
votes
0answers
26 views

What's the correct way of implementation of cost function and gradient function in logistical regression after regularisation?

This is the cost function of logistic regression: which i could implement correctly, with the code : ...
1
vote
3answers
119 views

What is the point of getting rid of overfitting?

I'm having trouble understanding why I would use dropout, regularization, data augmentation, etc to get rid of overfitting in the first place. I get that if your model is too large or data is too ...
0
votes
0answers
807 views

What are best activation and regularization method for LSTM?

In Keras there are: ...
2
votes
1answer
226 views

Square Root Regularization and High Loss

I am testing out square root regularization (explained ahead) in a pytorch implementation of a neural network. Square root regularization, henceforth l1/2, is just like l2 regularization, but instead ...
0
votes
0answers
62 views

Running into strange errors when using glmnet and generating graphs

I developed a very simple Ridge Regression with glmnet, the R package. When I use the plot.glmnet() function I encounter strange errors: ...
1
vote
1answer
50 views

neural networks error function: is global minimum desirable?

In "Elements of statistical learning" page 395 the authors state that, relative to R(θ), the regression/classification error function in a neural network such as a multi layer perceptron: Typically ...
-1
votes
1answer
94 views

Regularization: global or layerwise?

Keras gives you the option to apply regularization differently to different layers. I mean, why not? Though when I first learned about neural nets (from ESL), I thought of it as a global parameter. ...
1
vote
1answer
90 views

Importing Excel format data into R/R Studio and using glmnet package?

I have no problem importing Excel formatted data into R/R Studio and use all other R packages that I use. But, when I want to use the glmnet package to develop a regularization model, I invariably ...
1
vote
0answers
139 views

Keras regularizers (kernel, bias and activity) vs tf.contrib.layers.apply_regularization

I have a DCGAN set up in tensorflow that is working well on the faces in the wild dataset. As an experiment, I tried using the same architecture in keras to better understand the difference in ...
2
votes
1answer
31 views

R package clogitL1 no longer available? [closed]

When I try to install clogitL1 on my work server I get ...