Questions tagged [regularization]

Inclusion of additional constraints (typically a penalty for complexity) in the model fitting process. Used to prevent overfitting / enhance predictive accuracy.

0
votes
0answers
13 views

Regularization loss - Why is that important to get unique weights solution

There's something that is bothering me in Regularization. There is one bug with the loss function when I just use for e.g. a multi-class SVM loss. Suppose that we have a dataset and a set of ...
1
vote
2answers
26 views

Difference between LASSO penalty in neural network and just LASSO regression

I wonder whether those two have any significant differences. I think in neural network, the lasso penalty put on the loss function makes the model simpler and introduces more sparsity by ...
2
votes
2answers
31 views

SVM behavior when regularization parameter equals 0

I read on this Wikipedia page the following about soft-margin SVM: "The parameter $λ$ determines the trade-off between increasing the margin size and ensuring that the $x_i$ lie on the correct ...
2
votes
1answer
27 views

Multiclass classification with high number of classes, high number of features and small sample size

I am working on a biology related dataset with over 300K features, and I only have about 5K samples. I want my model to classify many classes. For this problem in particular the class is age. Each age ...
1
vote
1answer
29 views

correct ML approach

I wanted to get your thoughts on a problem I have been facing. I have daily level product sales information (about 4 years). The sales are affected by the typical factors such as seasonality, day of ...
0
votes
0answers
16 views

what is the relationship between Penalties and Regularization?

i am learning the DeepLearningBook, in which chapter7 talks a lot about Penalties and Regularization. i would like to figure out the relationship between Penalties and Regularization. it seems that ...
0
votes
0answers
9 views

What's the correct way of implementation of cost function and gradient function in logistical regression after regularisation?

This is the cost function of logistic regression: which i could implement correctly, with the code : ...
1
vote
3answers
89 views

What is the point of getting rid of overfitting?

I'm having trouble understanding why I would use dropout, regularization, data augmentation, etc to get rid of overfitting in the first place. I get that if your model is too large or data is too ...
0
votes
0answers
53 views
2
votes
1answer
63 views

Square Root Regularization and High Loss

I am testing out square root regularization (explained ahead) in a pytorch implementation of a neural network. Square root regularization, henceforth l1/2, is just like l2 regularization, but instead ...
0
votes
0answers
51 views

Running into strange errors when using glmnet and generating graphs

I developed a very simple Ridge Regression with glmnet, the R package. When I use the plot.glmnet() function I encounter strange errors: ...
1
vote
1answer
30 views

neural networks error function: is global minimum desirable?

In "Elements of statistical learning" page 395 the authors state that, relative to R(θ), the regression/classification error function in a neural network such as a multi layer perceptron: Typically ...
0
votes
1answer
20 views

Regularization: global or layerwise?

Keras gives you the option to apply regularization differently to different layers. I mean, why not? Though when I first learned about neural nets (from ESL), I thought of it as a global parameter. ...
1
vote
1answer
47 views

Importing Excel format data into R/R Studio and using glmnet package?

I have no problem importing Excel formatted data into R/R Studio and use all other R packages that I use. But, when I want to use the glmnet package to develop a regularization model, I invariably ...
1
vote
0answers
74 views

Keras regularizers (kernel, bias and activity) vs tf.contrib.layers.apply_regularization

I have a DCGAN set up in tensorflow that is working well on the faces in the wild dataset. As an experiment, I tried using the same architecture in keras to better understand the difference in ...
1
vote
0answers
59 views

How does L1 Regularization work in lightGBM

From the paper, lightGBM does a subsampling according to sorted $|g_i|$, where $g_i$ is the gradient (for the loss function) at a data instance. My question is that, when the objective is L1 loss/...
2
votes
1answer
12 views

R package clogitL1 no longer available? [closed]

When I try to install clogitL1 on my work server I get ...
0
votes
0answers
17 views

Why do we reduce magnitude of the coefficient in regression

Why do we reduce the magnitude of the coefficient in regression? how does it help the model?
1
vote
1answer
18 views

Make embedding more Gaussian-like

I am trying to train a neural network to find a mapping(embedding) to a lower dimensional space. I would like for my dataset, once mapped to the lower dimensional space, to appear gaussian-like ...
1
vote
1answer
32 views

Loss and Regularization inference

I'm building a Matrix Factorization model for MovieLens dataset with batch-wise training. Loss function for the batch: $$ L_{batch} = 1/|B|\sum_{(u,i)\in{B}}(r_{ui} - \mu - b_u - b_i - p_u^Tq_i)^2 + \...
2
votes
0answers
147 views

Regularization in Embedding models?

What is the best way to regularize latent embeddings, I have two solution in my mind but I'm not sure which one to use over other. In batch-wise training regularize over the whole embedding matrix, ...
0
votes
0answers
41 views

Matrix Factorization for Recommender Systems

Referring to the paper Matrix Factorization Techniques for Recommender Systems, Loss function for Matrix Factorization using bias terms is given as: $$ \min_{p, q, b}\sum_{(u,i)\in\kappa}(r_{ui} - \mu ...
0
votes
1answer
79 views

How does a Bayes regularization works?

I'm trying to get grasp of Bayes regularization algorithm. List of symbols 1st: $F$ - objective function $\gamma$ - regularization parameter $M$ - number od neural network weights $N$ - number of data ...
2
votes
0answers
80 views

Regularization term in Matrix Factorization

I'm trying to build a naive recommender system using latent factor model for MovieLens dataset. From the observed set of ratings I'm trying to build a model which will decompose the sparse matrix to N ...
0
votes
1answer
18 views

What affects the magnitude of lasso penalty of a feature?

Is there a way to intuitively tell if the lasso penalty for a particular feature will be small or large? Consider the following scenario: Imagine we use Lasso regression on a dataset of 100 features ...
3
votes
1answer
569 views

Which regularization in convolution layers (conv2D) [closed]

I am using Keras for a project. I would like to know if it makes any sense to add any kind of regularization components such as kernel, bias or activity regularization in convolutional layers i.e <...
3
votes
1answer
614 views

Regularization in simple math explained

I read a lot of articles online about how regularization works and most of them just show the equations with regularization terms but did not use example numbers to explain how the coefficient values ...
3
votes
1answer
185 views

Implemented early stopping but came across the error SGDClassifier: Not fitted error in sklearn

Below is the simpler implementation of early stopping which i came across the book and wanted to try it. ...
2
votes
1answer
51 views

Does Orange scale the data automatically for the linear regression with Ridge regularization

I'm using the linear regression tool with the Ridge regularization. To use the Ridge regularization, I have to scale the data first. Does Orange scale the data automatically? I can't find any ...
2
votes
0answers
169 views

Is regularization only for regression?

I am making a classification model. I understand that the regularization minimizes loss functions. Can i use regularization techniques to minimize loss function in classification if doing so is ...
2
votes
1answer
5k views

When should one use L1, L2 regularization instead of dropout layer, given that both serve same purpose of reducing overfitting?

In Keras, there are 2 methods to reduce over-fitting. L1,L2 regularization or dropout layer. https://keras.io/regularizers/ https://keras.io/layers/core/#dropout What are some situations to use L1,...
0
votes
1answer
575 views

Should highly correlated features be omitted before applying Lasso?

I would greatly appreciate if you could let me know whether I should omit highly correlated features before using Lasso logistic regression (L1) to do feature ...
12
votes
4answers
3k views

Why does adding a dropout layer in Keras improve machine learning performance, given that dropout suppresses some neurons from the model?

If removing some neurons results in a better performing model, why not use a simpler neural network with fewer layers and fewer neurons in the first place? Why build a bigger, more complicated model ...
1
vote
1answer
780 views

Is regularization included in loss history Keras returns?

I'm getting to know Keras. Right now, I'm testing with regularization and how to use them. Comparing the results of loss history for a training session with and without regularization, it seems to me ...
1
vote
0answers
46 views

Problems with Graphical Lasso

I'm trying to use the Graphical Lasso algorithm (more specifically the R package glasso) to find an estimated graph representing the connections between a set of nodes by estimating a precision matrix....
1
vote
0answers
26 views

Why should each layer's child network output be close to parent network's output for variance regularizer?

I am reading up on PEA (Pseudo ensemble agreement) regularizer. specificaly in the neural networks domain. It introduces the concept of perturbing the model a little and forcing the model to make ...
2
votes
1answer
156 views

Why don't we want Autoencoders to perfectly represent their training data?

From Ian Goodfellow's Deep Learning Book: If an autoencoder succeeds in simply learning to set g(f(x)) = x everywhere, then it is not especially useful. Instead, ...
1
vote
1answer
52 views

What is the intuition behind Ridge Regression and Adapting Gradient Descent algorithms?

So I was going through Adaptive Gradient Descent, and learning the intuition behind it: optimizing the learning algorithm, and getting the model to converge faster. The way AdaGrad does this, is by ...
1
vote
1answer
78 views

Loss for CNN decreases and settles but training accuracy does not improve

I am training a CNN with 2 conv layers 2 Relu and max pooling and 2 FC layers the last of which has only 2 units since it's a binary classification problem. The images are spatio-temporal continuous, ...
3
votes
1answer
97 views

GANs and grayscale imagery colorization

I am currently studying colorization of grayscale satellite imagery as part of my Master's internship. After looking for various machine learning techniques, I quickly decided to go for deep learning, ...
0
votes
1answer
148 views

Ridge and Lasso Regularization

Recently, I started working on Ridge and Lasso regularization for Linear and Logistic Regression. My doubts are given below: Is the penalty the same (by same proportion) for all the coefficients or ...
1
vote
1answer
54 views

Should I update my regularisation L1 and L2 regularisation parameters in online setting?

I have been working on online learning for a few weeks now, especially with Vowpal Wabbit and logistic regression. My understanding of the online learning algorithms and the problem is alright but I ...
2
votes
1answer
84 views

Dropout in other machine learning models

Dropout is a widely used technique in deep learning. Dropout was built for neural networks, but I wonder if other prediction models can use this idea as well as a regularizer. Do you know of any ...
6
votes
1answer
1k views

Dropout vs weight decay

Dropout and weight decay are both regularization techniques. From my experience, dropout has been more widely used in the last few years. Are there scenarios where weight decay shines more than ...
1
vote
1answer
21 views

Point of dropping weights in mini batch for purpose of regularization

I have been reading "drop" is a method to regularize model better. It's purpose is to update only some % of weights in backprop and it helps you to not over fit the model. But I am wondering, is this ...
3
votes
1answer
33 views

Can the 'bin size' in a histogram be thought of as a regularity constraint?

When thinking about a histogram as an estimate of the density function, is it reasonable to think of the bin size as a parameter that constrains the local structure of that function? Also, is there a ...
1
vote
1answer
435 views

trying to decrease overfitting with regularisation in CNN

I am doing transfer learning by retraining the publicly available inception layer, without regularisation here are my initial parameters and results: ...
1
vote
0answers
201 views

Concrete Dropout for Recurrent Neural Networks (Keras)

I would like to use the Concrete Dropout Framework from GAL in application to recurrent neural networks. There is a great paper about it and the implementation can be found on the website (Thank you ...
1
vote
0answers
83 views

Custom regularisation for logistics regression

My understanding of l2 regularisation: Weights of the model are assumed to have a prior guassian distribution centered around 0. Then MAP estimate over data adds an extra penalty in cost function. ...
6
votes
1answer
524 views

Weight decay in neural network

I have been reading through this book and am trying to do the exercises. The problem is "Connecting regularization and the improved method of weight initialization" part 3. We have to use a heuristic ...