Questions tagged [dropout]

Dropout is a technique to reduce overfitting during the training phase of a neural network.

Filter by
Sorted by
Tagged with
0 votes
2 answers
46 views

Dropout and BatchNorm decrease speed of learning

Experimenting with the cifar10 dataset and faced with strange behavior when Dropout and BatchNorm don't help at all. As I get: Dropout - freezing some of the weights which helps us to prevent ...
kirsanv43's user avatar
  • 103
0 votes
1 answer
36 views

Better results when adding a dropout layer before a single layer classifier - counter intuitive result

I am working on an multi-class image classification problem (with 9 classes), i am using a pretrained DenseNet121 (on ImageNet), i'm using Keras. i am using densenet as a feature extractor, with a ...
user062's user avatar
0 votes
0 answers
112 views

Regularization in CNN change training accuracy but not validation accuracy

I have a question regarding regularization in convolutional neural networks. So I'm building a CNN for image classification and I've come across something I don't understand. Without using any ...
ducksnack's user avatar
3 votes
1 answer
3k views

How can I Implement Dropout in SciKit-Learn?

I am working on an air-gapped PC and can only access the SciKit-Learn machine learning library. Unfortunately, its MLPClassifier doesn't include a Dropout ...
Connor's user avatar
  • 617
0 votes
1 answer
81 views

How does dropout behave like model averaging?

It is claimed Srivastava, Hinton, et al. that "dropout can be effectively applied in the hidden layers as well and that it can be interpreted as a form of model averaging" and that "...
Jack G's user avatar
  • 1
1 vote
0 answers
347 views

Dropout and LSTM layers

Consider the following code: ...
AAA's user avatar
  • 145
5 votes
1 answer
375 views

In neural networks, is applying dropout the same as zeroing random neurons?

Is applying dropout equivalent to zeroing output of random neurons in each mini-batch iteration and leaving rest of forward and backward steps in back-propagation unchanged? I'm implementing network ...
Qbik's user avatar
  • 195
0 votes
1 answer
45 views

The idea behind Dropout

I just need to check my understanding regarding the dropout regularization technique. According to my understanding, neurons can interact with each other in a way that they can fix each other's ...
John adams's user avatar
0 votes
1 answer
42 views

What is the problem that causes overfitting in the code?

** ...
Esra Ibra's user avatar
1 vote
1 answer
77 views

Using batchnorm and dropout simultaneously?

I am a bit confused about the relation between terms "Dropout" and "BatchNorm". As I understand, Dropout is regularization technique, which is using only during training. ...
AlexM's user avatar
  • 23
0 votes
1 answer
1k views

How is the validation set processed in PyTorch?

Say, one uses the MNIST dataset and splits the provided training data of size 60,000 into a training set (50,000) and a validation set (10,000). The provided test data of size 10,000 is used as the ...
Ronquam's user avatar
0 votes
1 answer
266 views

Lower training accuracy than testing accuracy (MLP/Dropout)

I am working on a problem of multi-class classification by MLP. I have set dropout to each middle layer. Now I observe the training accuracy is around 10% less than ...
hola's user avatar
  • 103
0 votes
1 answer
85 views

Can I apply Dropout In layers other than Fully Connected layers in CNN

I have read and seen that in CNN we apply DROPOUT layer between the FULLY CONNECTED layers to reduce overfitting. Can we also apply the dropout layer between the CONV layers and the POOL layers. I ...
Shiv's user avatar
  • 679
1 vote
2 answers
32 views

Structure of NN for input data with drop out

In financial markets, there is a simple problem of trading calendars varying across different countries. For example, Sweden observes Sweden National Day and Norway has Whit Monday. Typically, what ...
Attack68's user avatar
  • 171
1 vote
2 answers
1k views

Training a CNN on a large dataset

I am currently trying to build a CNN for around 100,000 images. There are 42 classes. I have used the default batch size of 32. This is how my model looks like: ...
eun ji's user avatar
  • 11
1 vote
0 answers
100 views

What is dropout in convolutional layers and how does that different from max-pooling-dropout?

When dropout is applied to fully connected layers some nodes will be randomly set to 0. It is unclear to me how dropout work with convolutional layers. If dropout is applied before the convolutions, ...
user570593's user avatar
0 votes
0 answers
256 views

Whats the difference between add.LSTM(num_hidden, droput=0.5) and add.Dropout(0.5) in Keras?

Could anyone please explain what is the difference between these two cases, specified in the title. I believe I am not the only one who is confused. I have read that it is preferrable to add Dropout ...
été's user avatar
  • 19
1 vote
0 answers
46 views

GAN training the average of my train data

I have been training a GAN with 1D convolutional layers on sinus functions. However if I start varying my sinus (random amplitude for example), the model generates only the average of the random range....
Casper Hogenboom's user avatar
4 votes
1 answer
142 views

Bayesian regularization vs dropout for basic ann

Does it make sense conceptually to apply dropout to an artificial neutral network while also applying bayesian regularization? On one hand I would think that technically this should work just fine, ...
Bram's user avatar
  • 143
1 vote
1 answer
689 views

Multiply weights after using dropout in training - PyTorch

I have a Pytorch regression model as follows: ...
Osama Dar's user avatar
  • 599
1 vote
1 answer
368 views

Improving Accuracy of the Deep Learning Model

In my current project, I have only 647 rows (500 for training and 147 for testing) and I have applied the Keras Sequential model using the following code: ...
Saurabh Chauhan's user avatar
2 votes
1 answer
597 views

Monte Carlo Dropout as Uncertainty predection

I am pretty new to Python and this board so I am not sure, if I am at the right place for my question since it doesn't include any code. If not so, please give my a hint for a better way/place to ask. ...
VomDorfe's user avatar
2 votes
0 answers
43 views

Intuitive reasoning behind inverted dropout in neural networks

I'm going through the deeplearning.ai course on Coursera and am trying to understand the intuitive reasoning behind inverted dropout in neural networks. Based on the lecture, my understanding is as ...
Shirish Kulhari's user avatar
0 votes
1 answer
41 views

Is generalizing a model, then removing the generalization good for FFNNs?

If one is training a basic FFNN (Feed-Forward Neural Network), one would apply regularizations like dropout, l1, l2 and gaussian noise, so that the model is robust and gives better results for unseen ...
ѕняєє ѕιиgнι's user avatar
4 votes
2 answers
1k views

Dropout in a CNN vs Dropout in a FCNN

In the PyTorch nn module there are 2 types of dropouts: A normal Dropout - During training, randomly zeroes some of the elements of the input tensor with ...
DuttaA's user avatar
  • 793
1 vote
0 answers
150 views

Is applying simultaneous K Fold Cross Validation and Drop out possible?

Well, it might seem ridiculous but I was just thinking whether it is possible to have these two methods simultaneously or not. I ran the code and faced an error, but in theory it doesn't seem ...
Fatemeh Asgarinejad's user avatar
2 votes
0 answers
643 views

Measuring uncertainty in an LSTM network using dropout in keras/tensorflow

I've created a simple LSTM network for testing ...
LMP's user avatar
  • 21
3 votes
1 answer
1k views

LSTM: Converting to Bayesian Deep Neural Network

Starting from Yarin Gal's research paper on using Dropout as a Bayesian Approximation (https://arxiv.org/pdf/1506.02142.pdf), I am trying to apply this concept to my Sequence Prediction model. My ...
Mohamed's user avatar
  • 31
4 votes
2 answers
3k views

how to apply MC dropout to an LSTM network keras

I have a simple LSTM network developped using keras: ...
khaoula's user avatar
  • 41
8 votes
1 answer
21k views

How does dropout work during testing in neural network?

The below paragraph is picked from the textbook Hands-On Machine Learning with sci-kit learn & Tensorflow. I couldn't understand what the author is trying to ...
James K J's user avatar
  • 457
20 votes
1 answer
18k views

What is Monte Carlo dropout?

I understand how to use MC dropout from this answer, but I don't understand how MC dropout works, what its purpose is, and how it differs from normal dropout.
Arka Mallick's user avatar
3 votes
1 answer
1k views

Dropout in Deep Neural Networks

I was reading a paper published on Dropout. What I find difficulty in understanding that, In the training phase, a unit is present with a probability $p$ and not present with a probability $1-p$. In ...
Akash Dubey's user avatar
6 votes
1 answer
3k views

What does SpatialDropout1D() do to output of Embedding() in Keras?

Keras model looks like this ...
GeorgeOfTheRF's user avatar
15 votes
2 answers
12k views

Why should we use (or not) dropout on the input layer?

People generally avoid using dropout at the input layer itself. But wouldn't it be better to use it? Adding dropout (given that it's randomized it will probably end up acting like another regularizer)...
Aditya's user avatar
  • 2,460
15 votes
2 answers
27k views

Dropout on which layers of LSTM?

Using a multi-layer LSTM with dropout, is it advisable to put dropout on all hidden layers as well as the output Dense layers? In Hinton's paper (which proposed ...
BigBadMe's user avatar
  • 740
1 vote
1 answer
229 views

How to interpret a drastic accuracy loss while training a neuronal net (CNN)?

How can one interpret a drastic accuracy loss after ~38 epochs? Maybe more dropout should be added to the CNN network? (x-axis shows the number of epochs)
Predicted Life's user avatar
29 votes
2 answers
23k views

When should one use L1, L2 regularization instead of dropout layer, given that both serve same purpose of reducing overfitting?

In Keras, there are 2 methods to reduce over-fitting. L1,L2 regularization or dropout layer. What are some situations to use L1,L2 regularization instead of dropout layer? What are some situations ...
user781486's user avatar
  • 1,335
16 votes
5 answers
8k views

Why does adding a dropout layer improve deep/machine learning performance, given that dropout suppresses some neurons from the model?

If removing some neurons results in a better performing model, why not use a simpler neural network with fewer layers and fewer neurons in the first place? Why build a bigger, more complicated model ...
user781486's user avatar
  • 1,335
2 votes
1 answer
2k views

What is coadaptation of neurons in neural networks?

Looking for a bare minimum example (3 hidden units only maybe?) for what weights of a neural network with heavily coadapted weights would look like and showcase why they are bad. Also, how is ...
figs_and_nuts's user avatar
1 vote
0 answers
29 views

Dropout onto pre-weighted vs onto pre-activated vector?

For any layer in my neural net, should I apply dropout onto an entering vector, or on the pre-activated vector? In other words: $$\vec q=W\cdot \vec x$$ $$\vec h = activate(drop(\vec q))$$ or: $$\...
Kari's user avatar
  • 2,706
3 votes
1 answer
1k views

Dropout on inputs instead on outputs = DropConnect?

Is dropping out parts of the Input vector better than dropping out parts of the Output vector? The latter literally makes this same neuron invisible to any further layers. On the contrary, ignoring ...
Kari's user avatar
  • 2,706
2 votes
1 answer
181 views

Dropout in other machine learning models

Dropout is a widely used technique in deep learning. Dropout was built for neural networks, but I wonder if other prediction models can use this idea as well as a regularizer. Do you know of any ...
David Masip's user avatar
  • 6,031
7 votes
1 answer
6k views

Dropout vs weight decay

Dropout and weight decay are both regularization techniques. From my experience, dropout has been more widely used in the last few years. Are there scenarios where weight decay shines more than ...
David Masip's user avatar
  • 6,031
4 votes
2 answers
4k views

Regularization - Combine drop out with early stopping

I'm building a RNN (recurrent neural network) with LSTM cells. I'm using time series to perform anomaly detection. When training my RNN I'm using a dropout of 0.5 and I'm early stopping with a ...
Mike Evers's user avatar
2 votes
2 answers
3k views

Dropout Decreases Test and Train Accuracy in one layer LSTM in Pytorch

I have a one layer lstm with pytorch on Mnist data. I know that for one layer lstm dropout option for lstm in pytorch does not operate. So, I have added a drop out at the beginning of second layer ...
Hadi Gharibi's user avatar
1 vote
0 answers
345 views

Concrete Dropout for Recurrent Neural Networks (Keras)

I would like to use the Concrete Dropout Framework from GAL in application to recurrent neural networks. There is a great paper about it and the implementation can be found on the website (Thank you ...
Torben.'s user avatar
  • 259
2 votes
1 answer
273 views

Do we need to increase training data size when increasing dropouts?

I am using a fully connected feed forward neural network built using keras for text classification. It consists of 3 hidden layer. I am planning to add a dropout layer after each hidden layer to ...
anas17in's user avatar
3 votes
1 answer
1k views

Probability of dropout growth

In the DNN literature, is there analysis or a term on a dropout ratio (oppositely-)proportional to the depth of a layer? By intuition, I'd like to dropout fewer neurons on the layers next to the ...
0xmax's user avatar
  • 373
1 vote
1 answer
133 views

Dropout implies stochastic descent?

The question is very simple, yet I can't find a quick confirmation on the web. It might seem obvious - by design, will Dropout always result in stochastic-looking gradient descent? (SGD) I've built ...
Kari's user avatar
  • 2,706
0 votes
1 answer
88 views

Dropout dividing by compensation term = overshoots the result?

When applying dropout mask, why is it acceptable to divide the resulting state by the percentage of survived neurons? I understand that it's to prevent signal from dying out. But I've done the test, ...
Kari's user avatar
  • 2,706