All Questions
Tagged with regularization deep-learning
25 questions
0
votes
0
answers
49
views
Using Embedding For Regularization
Is using embeddings for regularization a valid practice?
My reasoning for that is that encoding training/tests datasets into smaller vectors would allow a smaller network with fewer parameters and ...
1
vote
1
answer
1k
views
Hyperparameter Tuning vs Regularization
While designing the architecture of a Neural Network, should I consider adding regularization (like Dropout, L1/L2, etc.) even after optimizing the problem using Hyperparameter Tuning? What should be ...
- 87
3
votes
1
answer
787
views
Why is l1 regularization rarely used comparing to l2 regularization in Deep Learning?
l1 regularization increases sparsity, so unimportant weights are decreased closer to 0. In Deep Learning models, the input usually consists of thousands or millions of features/pixels, and the network ...
- 131
1
vote
0
answers
32
views
why use one regularisation technique over another?
why should I prefer L1 over L2, in fully-connected-layer or convolution?
why use dropout between 2 layers, when there is the option of regularising a layer(or both) with something like L1 or L2? and ...
2
votes
1
answer
54
views
Approximation of long sequence of layers by one layer
Consider the following situation : there is a deep neural network with a lot of layers, and in order to speed up the inference or for regularization purposes one would like to reduce the complexity of ...
2
votes
1
answer
1k
views
Is it better to use separately regularization methods for Neural Networks (L2/L1 & Dropout)
I have been exploring different regularization approaches and observed the most common to be using either Dropout Layers or L1/L2 Regularization. I have seen many debates of whether it is of interest ...
3
votes
2
answers
693
views
When I add regularization like L1,L2 , do I need more epochs to properly train my model?
When I add regularization techniques in my model like L1 or L2 do i need more epochs to properly converge my model.
...
- 709
2
votes
1
answer
5k
views
How to reduce overfitting in a pre-trained network
I have a custom dataset with 10 classes and I am using a pre-trained resnet18 model from torch-vision. I can clearly see it's over-fitting because: the model is trained for 75 epochs with a batch size ...
- 143
2
votes
0
answers
425
views
How to interpret curve of regularization loss during CNN training?
I am fine-tuning a single shot detector (SSD) in tensorflow object detection api.
I didn't freeze the backbone (mobilenet), I programmed the learning rate to go from e-3 to e-4 to e-5.
In the paper ...
- 31
3
votes
3
answers
859
views
Should you turn off label smoothing when validating?
As the subject says. On one hand, the answer should be yes because label smoothing is a regularization feature and how can you know if it improves performance without turning it off? On the other hand,...
1
vote
2
answers
460
views
Should the lambda for L1 norm regularizer inversely be proportional to the number of trainable weights?
Say I want to implement Conv2D in keras and for each Conv2D layer, if I apply 20 filters of [2,3] filter on an input with depth of 10, then there will be 20*(2*3*10+1) = 1220 trainable weights.
the ...
- 33
1
vote
2
answers
454
views
Why bias is not considering in Regularization?
Most of the Regularization (L1, L2 ) techniques focused mostly on the weight term only .But Regularization is not considering Bias.From my understanding large bias doesn’t make a neuron sensitive ...
- 121
2
votes
2
answers
632
views
How does L1 regularization make low-value features more zero than L2?
Below formulas, L1 and L2 regularization
Many experts said that L1 regularization makes low-value features zero because of constant value. However, I think that L2 regularization could also make zero ...
- 125
1
vote
3
answers
205
views
What is the point of getting rid of overfitting?
I'm having trouble understanding why I would use dropout, regularization, data augmentation, etc to get rid of overfitting in the first place. I get that if your model is too large or data is too ...
- 191
1
vote
2
answers
235
views
Make embedding more Gaussian-like
I am trying to train a neural network to find a mapping(embedding) to a lower dimensional space. I would like for my dataset, once mapped to the lower dimensional space, to appear gaussian-like ...
- 11
16
votes
5
answers
9k
views
Why does adding a dropout layer improve deep/machine learning performance, given that dropout suppresses some neurons from the model?
If removing some neurons results in a better performing model, why not use a simpler neural network with fewer layers and fewer neurons in the first place? Why build a bigger, more complicated model ...
- 1,445
2
votes
1
answer
709
views
Why don't we want Autoencoders to perfectly represent their training data?
From Ian Goodfellow's Deep Learning Book:
If an autoencoder succeeds in simply learning to set g(f(x)) = x everywhere, then it is not especially useful. Instead, ...
- 121
4
votes
1
answer
260
views
GANs and grayscale imagery colorization
I am currently studying colorization of grayscale satellite imagery as part of my Master's internship. After looking for various machine learning techniques, I quickly decided to go for deep learning, ...
- 43
8
votes
1
answer
7k
views
Dropout vs weight decay
Dropout and weight decay are both regularization techniques. From my experience, dropout has been more widely used in the last few years. Are there scenarios where weight decay shines more than ...
- 6,106
1
vote
1
answer
1k
views
trying to decrease overfitting with regularisation in CNN
I am doing transfer learning by retraining the publicly available inception layer, without regularisation here are my initial parameters and results:
...
- 161
6
votes
1
answer
1k
views
Weight decay in neural network
I have been reading through this book and am trying to do the exercises.
The problem is "Connecting regularization and the improved method of weight initialization" part 3. We have to use a heuristic ...
- 71
2
votes
1
answer
95
views
How to think about prediction error that is not convex in hyperparameter, or over the course of training
Take the following case of a hyperparameter and prediction error:
Imagine that the hyperparameter is a L2 penalty or a dropout rate -- something that we think that should have a single sweet spot -- ...
- 499
1
vote
1
answer
242
views
Should I set higher dropout prob if there are plenty of data?
I have some excessive amount of data for the size of NN I am able to teach in a reasonable time.
If I feed all the data into the network it stops learning at some point and a resulting model shows ...
- 281
3
votes
1
answer
7k
views
What's the best way to tune the regularization parameter in neural nets
I'm tuning the regularization parameter of a neural net (L2 regularization) using a grid. Starting with values 0.0005, 0.005, 0.05, 0.5, 5. Then if ...
- 231
4
votes
1
answer
1k
views
Recommendations and Missing Data in Deep Learning
In this research paper, it is discussed how to combine deep learning with wide (shallow) learning to achieve both generalisation and the ability to learn correlation/association rules.
The input ...
- 85