Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [regularization]

The tag has no usage guidance.

0
votes
0answers
12 views

Different Criterions for Test MSE

I tried to reproduce elastic net simulation results table on the page 313 of elastic net paper. The authors stated that they simulated 50 datasets consisting of 20/20/200 observations for train/...
3
votes
1answer
92 views

Regularization in simple math explained

I read a lot of articles online about how regularization works and most of them just show the equations with regularization terms but did not use example numbers to explain how the coefficient values ...
3
votes
1answer
29 views

Implemented early stopping but came across the error SGDClassifier: Not fitted error in sklearn

Below is the simpler implementation of early stopping which i came across the book and wanted to try it. ...
2
votes
1answer
25 views

Does Orange scale the data automatically for the linear regression with Ridge regularization

I'm using the linear regression tool with the Ridge regularization. To use the Ridge regularization I have to scale the data first. Does Orange scale the data automatically? I can't find any ...
2
votes
0answers
37 views

Is regularization only for regression?

I am making a classification model. I understand that the regularization minimizes loss functions. Can i use regularization techniques to minimize loss function in classification if doing so is ...
0
votes
0answers
26 views

Can I make some nodes in keras immune to regularization?

I'm building an architecture to correspond with a partially linear model: $$ y = \alpha D + \mathbf{X}\beta + f(\mathbf{Z}) + \epsilon $$ where the $f$ is a neural net, and keras/tensorflow trains ...
2
votes
1answer
161 views

When should one use L1, L2 regularization instead of dropout layer, given that both serve same purpose of reducing overfitting?

In Keras, there are 2 methods to reduce over-fitting. L1,L2 regularization or dropout layer. https://keras.io/regularizers/ https://keras.io/layers/core/#dropout What are some situations to use L1,...
0
votes
1answer
195 views

Should highly correlated features be omitted before applying Lasso?

I would greatly appreciate if you could let me know whether I should omit highly correlated features before using Lasso logistic regression (L1) to do feature ...
10
votes
4answers
2k views

Why does adding a dropout layer in Keras improve machine learning performance, given that dropout suppresses some neurons from the model?

If removing some neurons result in a better performing model, why not use a simpler neural network with fewer layers, fewer neurons in the first place? Why build a bigger, more complicated model in ...
0
votes
1answer
105 views

Is regularization included in loss history Keras returns?

I'm getting to know Keras. Right now, I'm testing with regularization and how to use them. Comparing the results of loss history for a training session with and without regularization, it seems to me ...
1
vote
0answers
17 views

Problems with Graphical Lasso

I'm trying to use the Graphical Lasso algorithm (more specifically the R package glasso) to find an estimated graph representing the connections between a set of nodes by estimating a precision matrix....
1
vote
0answers
25 views

Why should each layer's child network output be close to parent network's output for variance regularizer?

I am reading up on PEA (Pseudo ensemble agreement) regularizer. specificaly in the neural networks domain. It introduces the concept of perturbing the model a little and forcing the model to make ...
2
votes
1answer
41 views

Why don't we want Autoencoders to perfectly represent their training data?

From Ian Goodfellow's Deep Learning Book: If an autoencoder succeeds in simply learning to set g(f(x)) = x everywhere, then it is not especially useful. Instead, ...
1
vote
1answer
31 views

What is the intuition behind Ridge Regression and Adapting Gradient Descent algorithms?

So I was going through Adaptive Gradient Descent, and learning the intuition behind it: optimizing the learning algorithm, and getting the model to converge faster. The way AdaGrad does this, is by ...
0
votes
0answers
13 views

Mathematics proof for L1 as parameter reduction

We know that L1 regularization is a way to do the parameter reduction, but I am wondering what's the mathematical proof of this? Thanks!
1
vote
1answer
43 views

Loss for CNN decreases and settles but training accuracy does not improve

I am training a CNN with 2 conv layers 2 Relu and max pooling and 2 FC layers the last of which has only 2 units since it's a binary classification problem. The images are spatio-temporal continuous, ...
3
votes
1answer
58 views

GANs and grayscale imagery colorization

I am currently studying colorization of grayscale satellite imagery as part of my Master's internship. After looking for various machine learning techniques, I quickly decided to go for deep learning, ...
0
votes
1answer
58 views

Ridge and Lasso Regularization

Recently, I started working on Ridge and Lasso regularization for Linear and Logistic Regression. My doubts are given below: Is the penalty the same (by same proportion) for all the coefficients or ...
0
votes
0answers
54 views

which regression model is better for one-hot encoding features (words) value

I am using the regression model to extract the importance of words in the texts with click rate value. I used the words one-hot encoding value to build features. For example, extracted all words of ...
1
vote
1answer
50 views

Should I update my regularisation L1 and L2 regularisation parameters in online setting?

I have been working on online learning for a few weeks now, especially with Vowpal Wabbit and logistic regression. My understanding of the online learning algorithms and the problem is alright but I ...
2
votes
1answer
58 views

Dropout in other machine learning models

Dropout is a widely used technique in deep learning. Dropout was built for neural networks, but I wonder if other prediction models can use this idea as well as a regularizer. Do you know of any ...
5
votes
1answer
543 views

Dropout vs weight decay

Dropout and weight decay are both regularization techniques. From my experience, dropout has been more widely used in the last few years. Are there scenarios where weight decay shines more than ...
0
votes
0answers
14 views

Regularised Model performing worse in sparse data

I trained a linear model and used the Lasso to regularise but I am finding for some of the smaller categories I am worsening my prediction. I know the data is sparse there and susceptible to a lot of ...
1
vote
1answer
17 views

Point of dropping weights in mini batch for purpose of regularization

I have been reading "drop" is a method to regularize model better. It's purpose is to update only some % of weights in backprop and it helps you to not over fit the model. But I am wondering, is this ...
3
votes
1answer
27 views

Can the 'bin size' in a histogram be thought of as a regularity constraint?

When thinking about a histogram as an estimate of the density function, is it reasonable to think of the bin size as a parameter that constrains the local structure of that function? Also, is there a ...
1
vote
1answer
172 views

trying to decrease overfitting with regularisation in CNN

I am doing transfer learning by retraining the publicly available inception layer, without regularisation here are my initial parameters and results: ...
0
votes
0answers
48 views

Regularising a Logistic Regression causes a class imbalance

I am trying to train a multi-class logistic regression model. The dataset is an embedding vector of length 500, which is used as features, and the target is 5 classes 0-4. This model is used in a ...
1
vote
0answers
138 views

Concrete Dropout for Recurrent Neural Networks (Keras)

I would like to use the Concrete Dropout Framework from GAL in application to recurrent neural networks. There is a great paper about it and the implementation can be found on the website (Thank you ...
1
vote
0answers
62 views

Custom regularisation for logistics regression

My understanding of l2 regularisation: Weights of the model are assumed to have a prior guassian distribution centered around 0. Then MAP estimate over data adds an extra penalty in cost function. ...
0
votes
0answers
98 views

Rebuild logistic regression model without regularisation sklearn

I built a logistic regression model using sklearn on 80+ features. After regularisation (L1) there were 10 non-zero features left. I want to turn this model into a production model but I don't want to ...
6
votes
1answer
361 views

Weight decay in neural network

I have been reading through this book and am trying to do the exercises. The problem is "Connecting regularization and the improved method of weight initialization" part 3. We have to use a heuristic ...
0
votes
0answers
44 views

Using two or more regularization method at the same time

There are two well-known regularization methods for fighting overfitting: 1) Using L1/L2 Regularization(Weight Decay) 2) Adding Dropout The question is: when/where it could be useful to utilize ...
2
votes
1answer
40 views

How to think about prediction error that is not convex in hyperparameter, or over the course of training

Take the following case of a hyperparameter and prediction error: Imagine that the hyperparameter is a L2 penalty or a dropout rate -- something that we think that should have a single sweet spot -- ...
1
vote
1answer
49 views

Dropout without the averaging

The final step in dropout regularization is to multiply the weights by the dropout probability. This is motivated by analogy to bagging: averaging the weights of multiple nets. But it isn't truly that ...
1
vote
0answers
87 views

Why is there no implementation of LARS in sklearn's LogisticRegression?

Following up on Efficient L1 Regularized Logistic Regression - Su-In Lee, Honglak Lee, Pieter Abbeel and Andrew Y. Ng Why is there no option for LARS as a solver for L1 penalized LogisticRegression ...
-1
votes
1answer
888 views

Which regularizer to use to get a sparse set of regression parameters?

I am doing a regression and I want to use the regularizer that will be the most useful to get a sparse set of parameters. Which regularizer should I use ? Cardinality? maximum value ? Sum of absolute ...
1
vote
1answer
100 views

Support vector machine margin term, norm or norm squared?

$$ \lambda||\hat w||^2 +(1/n)\sum max(0,1-y_i(\hat w \hat x_i -b)) $$ we know that $2/||\hat w||$ is the width of the margin. The second term penalizes a misclassified point for how far away it ...
10
votes
2answers
2k views

Why using L1 regularization over L2?

Conducting a linear regression model using a loss function, why should I use $L_1$ instead of $L_2$ regularization? Is it better at preventing overfitting? Is it deterministic (so always a unique ...
3
votes
1answer
508 views

Shouldn't L2 regularization be normalized for the number of nodes in a layer?

I'm naively thinking it would be better to normalize L2 regularization to the number of elements in a tensor, but I don't see anyone doing that. What am I missing? I'd say in a multi layer fully ...
1
vote
1answer
87 views

Should I set higher dropout prob if there are plenty of data?

I have some excessive amount of data for the size of NN I am able to teach in a reasonable time. If I feed all the data into the network it stops learning at some point and a resulting model shows ...
1
vote
0answers
760 views

Lasso implementation in Python

I am working on this course on Machine Learning 2012 from UBC (CPSC 340). I am stuck on a Homework code problem which shows the following RuntimeError in ...
1
vote
0answers
55 views

Selection of co related variables for ridge regression

I'm a newbie to machine learning. I'm working on a dataset to predict a target variable in terms of the independent variables. In the dataset the independent variables few are very highly correlated . ...
1
vote
3answers
442 views

Is LASSO regression implemented in Statsmodels?

I would love to use a linear LASSO regression within statsmodels, so to be able to use the 'formula' notation for writing the model, that would save me quite some coding time when working with many ...
3
votes
0answers
122 views

SVM regularization - minimizing margin?

I'm currently studying from Andrew Ng's Stanford handouts here (I'm at part 8). Now from what I gathered from before, all the time our goal was to minimize ||w||^2 so that we can maximize the margin. ...
3
votes
1answer
1k views

Why does dropout ruin my accuracy in CNN?

I've build a CNN in Tensorflow with 2 conv layers, 1 pool layer and 2 FC layers. When I don't use dropout I get 98% accuracy on training dataset and 90% on test dataset. But, when I do use dropout, I ...
1
vote
0answers
257 views

L2 regularization in caffe

I have a lasgane code. I want to create the same network using caffe. I could conver the network.But i need help with the hyperparameters in lasagne. The hyperparameters look like: ...
1
vote
1answer
40 views

How can I fix this “convex” problem ? Is it just a matter of overfitting?

I get some metrics on validation data while training a model , and in my case the they are : (0.25, 0.31, 0.46, 0.57, 0.65, 0.75, 0.77, 0.78, 0.84, 0.84, 0.85, 0.84, 0.84, 0.84, 0.82, 0.8, 0.8, 0....
6
votes
2answers
2k views

Convolutional Neural Network overfitting

I built a CNN to learn to classify EEG data (only about 4000 training examples, 2 classes, 50-50 class balance). Each training example is 64x512, with 5 channels each Ive tried to keep the network as ...
4
votes
1answer
3k views

Regularization practice with ANNs

I have learnt from some examples the existence of regularization option at ANNs (concretely, at Keras implementation). As far as I know, regularization in general is a kind of "penalty" on parameters ...
3
votes
1answer
1k views

What's the best way to tune the regularization parameter in neural nets

I'm tuning the regularization parameter of a neural net (L2 regularization) using a grid. Starting with values 0.0005, 0.005, 0.05, 0.5, 5. Then if ...