Questions tagged [regularization]

Inclusion of additional constraints (typically a penalty for complexity) in the model fitting process. Used to prevent overfitting / enhance predictive accuracy.

Filter by
Sorted by
Tagged with
0
votes
0answers
5 views

early-stopping changes final epoch in training each time

I am training a CNN built using transfer learning with a VGG16 network as pre-trained model, and in the training I am using early-stopping as regularization technique. I have run several time the ...
2
votes
2answers
173 views

How to build an overfitted network in order to increase performances

I am learning how to implement CNN, and searching on the internet I have found that a trick to design a good network is to first build it in such a way that it overfits, and then use regularization to ...
1
vote
1answer
21 views

How regularization helps to get rid of outliers?

I have heard regularization helps to get rid of outliers, how so? 'My intuition is, regularization shrinks parameter or even make it zero, and hence large value will have less effect on overall result'...
0
votes
0answers
8 views

What's intuition behind the activity regularizer? Any practical application?

I have read and understand that the activity regularizer is to operate on a neural layer output to make it smaller. However, I couldn't have any intuition for it and don't know why making output ...
1
vote
1answer
18 views

best way to regularize gradient boosting regressor?

i am testing gradient boosting regressor from sklearn for time series prediction on noisy data (currency markets). https://scikit-learn.org/stable/modules/generated/sklearn.ensemble....
1
vote
0answers
22 views

How can I regularize the output of a layer from scratch (without using Keras)?

I am trying to build a Convolutional Neural Network after reading notes from Stanford's cs231n course. I use ELU activation as activation function, and SoftMax as my classifier. Architecture is simple:...
0
votes
0answers
18 views

Should the LightGBM score match the regularization?

If I set the parameter objective to regression_l1 and set the metric to mean absolute error in ...
2
votes
2answers
40 views

If my model is overfitting the training dataset, does adding noise to training dataset help regularizing the machine learning model

I would like to know if this is a best practice or not. Can we add noise to the training data to help the model "fit less the training data"; as a result, hoping to generalize better on new unseen ...
0
votes
0answers
18 views

Using normal distribution data with high amplitude

I am using some data from a csv file to train a model which detects credit card fraud. The data set is from Kaggle and has 284,xxx samples each with 30 features. After some visualization in Python, ...
0
votes
0answers
13 views

Is this interpretation of spectral normalisation mathematically correct?

Hello everyone, this is my first post. I was thinking about the mathematical interpretation for spectral normalization in neural networks the other day, and I came up with an explanation that feels ...
0
votes
0answers
12 views

Binary Matrix Factorization: Regularizer to encourage 1-0 matrices

I have the following problem: Given a (user, access rights) binary matrix, I need to find the best (user, role) x (role, access right) binary matrices to reconstruct the original matrix. The current ...
1
vote
0answers
22 views

Do we need to divide our gradients by batch size our we will use the sum (Mini batch GSD plus L2 Regularization)?

I am implementing L2 regularization in C++ and I used mini batch GSD. Without L2, I was using sum of gradients during back propagation and I was not dividing my cost function by batch size. I was ...
0
votes
0answers
10 views

Regularizing Neural Network for deterministic function approximation

I'm training a neural network to learn a specific pricing function, which is entirely deterministic (i.e. same inputs always produce same outputs). The training occurs with 80 million data points from ...
0
votes
0answers
8 views

Does ridge regression reduce the coefficients of the variables all the way to zero at very high penalty

I have read in one article that ridge regression doesn't reduce the coefficients of the features to zero where as in some other article I read that it can reduce the coefficients to zero when a very ...
2
votes
3answers
91 views

Can ridge regression be used for feature selection?

I'm trying to figure out whether using Ridge Regression for regularization can be used to cause a more sparse hypothesis however to me it seems like ridge will never actually bring any coefficients to ...
2
votes
1answer
42 views

Quadratic approximation of L1 regularized cost function

I'm reading the Deep Learning book of Goodfellow, but I fail to see why minimization of (7.22) gives (7.23). I tried to compute the gradient w.r.t. the $w_{i}$ and set this to zero, but it doesn't ...
1
vote
0answers
23 views

High Variance on CNN

I'm using a shallow CNN for my current project [this one]. I have a training dataset consisting of 1000 samples and a test dataset of 400 samples. I'm using the test dataset to choose the best ...
2
votes
1answer
29 views

Why do we determine the values of λ in regularization as ln λ, such as ln λ=-18 instead of for example λ=0.3?

I'm studying Pattern Recognition and Machine Learning by Christopher Bishop. What I realized is, he defines values of λ as ln λ. For example: We see that, for a value of lnλ = −18, the over-fitting ...
1
vote
1answer
41 views

Why do we divide the regularization term by the number of examples in regularized logistic regression?

So this is the formula for the regularized logistic regression cost function: $x^{(i)}$ - the $i$'th training example $\theta_j$ - the parameter of the $j$'th feature $m$ - the number of training ...
4
votes
2answers
160 views

Why use regularization instead of decreasing the model

Regularization is used to decrease the capacity of a machine learning model to avoid overfitting. Why don't we just use a model with less capacity (e.g. decrease the number of layers). This would also ...
4
votes
2answers
729 views

L1 & L2 Regularization in Light GBM

This question pertains to L1 & L2 regularization parameters in Light GBM. As per official documentation: reg_alpha (float, optional (default=0.)) – L1 ...
1
vote
1answer
248 views

Light GBM Regressor, L1 & L2 Regularization and Feature Importances

I want to know how L1 & L2 regularization works in Light GBM and how to interpret the feature importances. Scenario is: I used LGBM Regressor with RandomizedSearchCV (cv=3, iterations=50) on a ...
4
votes
2answers
66 views

Why does Lasso behave “erratically” when the number of features is greater than the number of training instances?

From the book "Hands-on Machine Learning with Scikit-Learn and TensorFlow 2nd edition" chapter 4: In general, Elastic Net is preferred over Lasso since Lasso may behave erratically when the ...
1
vote
1answer
40 views

Improving Accuracy of the Deep Learning Model

In my current project, I have only 647 rows (500 for training and 147 for testing) and I have applied the Keras Sequential model using the following code: ...
1
vote
0answers
28 views

Version of Perceptron

If we change the $ywx<0$ condition (for performing update) to $ywx<1$ like in SVM (but without adding regularization to maximize the margin), is there any difference from the basic perceptron (...
1
vote
1answer
40 views

Why non-differentiable regularization lead to setting coefficients to 0?

The L2 regularization lead to minimize the values in the vector parameter. The L1 regularization lead to setting some coefficients to 0 in the vector parameter. More generally, I've seen that non-...
0
votes
0answers
26 views

Regularization loss - Why is that important to get unique weights solution

There's something that is bothering me in Regularization. There is one bug with the loss function when I just use for e.g. a multi-class SVM loss. Suppose that we have a dataset and a set of ...
1
vote
2answers
102 views

Difference between LASSO penalty in neural network and just LASSO regression

I wonder whether those two have any significant differences. I think in neural network, the lasso penalty put on the loss function makes the model simpler and introduces more sparsity by ...
3
votes
2answers
78 views

SVM behavior when regularization parameter equals 0

I read on this Wikipedia page the following about soft-margin SVM: "The parameter $λ$ determines the trade-off between increasing the margin size and ensuring that the $x_i$ lie on the correct ...
2
votes
1answer
52 views

Multiclass classification with high number of classes, high number of features and small sample size

I am working on a biology related dataset with over 300K features, and I only have about 5K samples. I want my model to classify many classes. For this problem in particular the class is age. Each age ...
1
vote
1answer
44 views

correct ML approach

I wanted to get your thoughts on a problem I have been facing. I have daily level product sales information (about 4 years). The sales are affected by the typical factors such as seasonality, day of ...
0
votes
0answers
16 views

what is the relationship between Penalties and Regularization?

i am learning the DeepLearningBook, in which chapter7 talks a lot about Penalties and Regularization. i would like to figure out the relationship between Penalties and Regularization. it seems that ...
0
votes
0answers
17 views

What's the correct way of implementation of cost function and gradient function in logistical regression after regularisation?

This is the cost function of logistic regression: which i could implement correctly, with the code : ...
1
vote
3answers
107 views

What is the point of getting rid of overfitting?

I'm having trouble understanding why I would use dropout, regularization, data augmentation, etc to get rid of overfitting in the first place. I get that if your model is too large or data is too ...
0
votes
0answers
458 views

What are best activation and regularization method for LSTM?

In Keras there are: ...
2
votes
1answer
152 views

Square Root Regularization and High Loss

I am testing out square root regularization (explained ahead) in a pytorch implementation of a neural network. Square root regularization, henceforth l1/2, is just like l2 regularization, but instead ...
0
votes
0answers
58 views

Running into strange errors when using glmnet and generating graphs

I developed a very simple Ridge Regression with glmnet, the R package. When I use the plot.glmnet() function I encounter strange errors: ...
1
vote
1answer
40 views

neural networks error function: is global minimum desirable?

In "Elements of statistical learning" page 395 the authors state that, relative to R(θ), the regression/classification error function in a neural network such as a multi layer perceptron: Typically ...
-1
votes
1answer
67 views

Regularization: global or layerwise?

Keras gives you the option to apply regularization differently to different layers. I mean, why not? Though when I first learned about neural nets (from ESL), I thought of it as a global parameter. ...
1
vote
1answer
62 views

Importing Excel format data into R/R Studio and using glmnet package?

I have no problem importing Excel formatted data into R/R Studio and use all other R packages that I use. But, when I want to use the glmnet package to develop a regularization model, I invariably ...
1
vote
0answers
120 views

Keras regularizers (kernel, bias and activity) vs tf.contrib.layers.apply_regularization

I have a DCGAN set up in tensorflow that is working well on the faces in the wild dataset. As an experiment, I tried using the same architecture in keras to better understand the difference in ...
2
votes
1answer
26 views

R package clogitL1 no longer available? [closed]

When I try to install clogitL1 on my work server I get ...
0
votes
0answers
25 views

Why do we reduce magnitude of the coefficient in regression

Why do we reduce the magnitude of the coefficient in regression? how does it help the model?
1
vote
1answer
22 views

Make embedding more Gaussian-like

I am trying to train a neural network to find a mapping(embedding) to a lower dimensional space. I would like for my dataset, once mapped to the lower dimensional space, to appear gaussian-like ...
1
vote
1answer
50 views

Loss and Regularization inference

I'm building a Matrix Factorization model for MovieLens dataset with batch-wise training. Loss function for the batch: $$ L_{batch} = 1/|B|\sum_{(u,i)\in{B}}(r_{ui} - \mu - b_u - b_i - p_u^Tq_i)^2 + \...
2
votes
0answers
255 views

Regularization in Embedding models?

What is the best way to regularize latent embeddings, I have two solution in my mind but I'm not sure which one to use over other. In batch-wise training regularize over the whole embedding matrix, ...
0
votes
0answers
51 views

Matrix Factorization for Recommender Systems

Referring to the paper Matrix Factorization Techniques for Recommender Systems, Loss function for Matrix Factorization using bias terms is given as: $$ \min_{p, q, b}\sum_{(u,i)\in\kappa}(r_{ui} - \mu ...
0
votes
1answer
413 views

How does a Bayes regularization works?

I'm trying to get grasp of Bayes regularization algorithm. List of symbols 1st: $F$ - objective function $\gamma$ - regularization parameter $M$ - number od neural network weights $N$ - number of data ...
2
votes
0answers
111 views

Regularization term in Matrix Factorization

I'm trying to build a naive recommender system using latent factor model for MovieLens dataset. From the observed set of ratings I'm trying to build a model which will decompose the sparse matrix to N ...
0
votes
1answer
20 views

What affects the magnitude of lasso penalty of a feature?

Is there a way to intuitively tell if the lasso penalty for a particular feature will be small or large? Consider the following scenario: Imagine we use Lasso regression on a dataset of 100 features ...