Questions tagged [dropout]

Dropout is a technique to reduce overfitting during the training phase of a neural network.

20 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
2
votes
0answers
18 views

Intuitive reasoning behind inverted dropout in neural networks

I'm going through the deeplearning.ai course on Coursera and am trying to understand the intuitive reasoning behind inverted dropout in neural networks. Based on the lecture, my understanding is as ...
2
votes
0answers
364 views

Measuring uncertainty in an LSTM network using dropout in keras/tensorflow

I've created a simple LSTM network for testing ...
2
votes
1answer
726 views

LSTM: Converting to Bayesian Deep Neural Network

Starting from Yarin Gal's research paper on using Dropout as a Bayesian Approximation (https://arxiv.org/pdf/1506.02142.pdf), I am trying to apply this concept to my Sequence Prediction model. My ...
1
vote
0answers
16 views

What is dropout in convolutional layers and how does that different from max-pooling-dropout?

When dropout is applied to fully connected layers some nodes will be randomly set to 0. It is unclear to me how dropout work with convolutional layers. If dropout is applied before the convolutions, ...
1
vote
0answers
18 views

Whats the difference between add.LSTM(num_hidden, droput=0.5) and add.Dropout(0.5) in Keras?

Could anyone please explain what is the difference between these two cases, specified in the title. I believe I am not the only one who is confused. I have read that it is preferrable to add Dropout ...
1
vote
0answers
32 views

GAN training the average of my train data

I have been training a GAN with 1D convolutional layers on sinus functions. However if I start varying my sinus (random amplitude for example), the model generates only the average of the random range....
1
vote
0answers
212 views

Monte Carlo Dropout as Uncertainty predection

I am pretty new to Python and this board so I am not sure, if I am at the right place for my question since it doesn't include any code. If not so, please give my a hint for a better way/place to ask. ...
1
vote
0answers
41 views

Is applying simultaneous K Fold Cross Validation and Drop out possible?

Well, it might seem ridiculous but I was just thinking whether it is possible to have these two methods simultaneously or not. I ran the code and faced an error, but in theory it doesn't seem ...
1
vote
0answers
20 views

Dropout onto pre-weighted vs onto pre-activated vector?

For any layer in my neural net, should I apply dropout onto an entering vector, or on the pre-activated vector? In other words: $$\vec q=W\cdot \vec x$$ $$\vec h = activate(drop(\vec q))$$ or: $$\...
1
vote
0answers
266 views

Concrete Dropout for Recurrent Neural Networks (Keras)

I would like to use the Concrete Dropout Framework from GAL in application to recurrent neural networks. There is a great paper about it and the implementation can be found on the website (Thank you ...
1
vote
0answers
105 views

Why isn't Maxout used in the state of the art models?

I have just read the paper from Ian Goodfellow et al. titled "Maxout Networks". It seems that the Maxout activation should be quite powerful, as it can approximate any convex function, i.e. Relu, ...
0
votes
2answers
43 views

Training a CNN on a large dataset

I am currently trying to build a CNN for around 100,000 images. There are 42 classes. I have used the default batch size of 32. This is how my model looks like: ...
0
votes
0answers
14 views

CNN Architecture comparison standards

I want to add comparison of accuracy section in my study report on CNN Architecture for a medical data. I have already added the comparison by VGG 16, AlexNet etc. Is it a standard to compare the ...
0
votes
0answers
12 views

Understanding usage of dropout in Keras

I would like to check if my understanding of how dropout layers should be used in Keras training is correct. I am training pretty simple MLP regression models: ...
0
votes
0answers
14 views

Concrete Dropout model unstable on regression tasks

Concrete Dropout as proposed by the author outputs the log variance which is used in calculating losses. Given that there are random parameter initializations in a NN, I am liable to get any number ...
0
votes
0answers
14 views

Batch Normalization when CNN with only 2 ConvLayer?

I wonder if it is a problem to use BatchNormalization when there are only 2 convolutional layers in a CNN. Can this have adverse effects on classification performance? Now I don't mean the training ...
0
votes
0answers
48 views

Dropout noise shape when applying it on a series

I am training a neural network based on the Deep Sets framework (https://arxiv.org/abs/1703.06114, https://arxiv.org/abs/1810.05165). The basis of this approach is that one has a series of input ...
0
votes
0answers
119 views

Why are deep ensembles and monte carlo dropout never used simultanuously in uncertainty estimation

In papers on this topic, I have seen deep ensembles being compared to dropout monte carlo. I was wondering why they are never used simultanously, since adding dropout monte carlo to every member of an ...
0
votes
1answer
21 views

Is generalizing a model, then removing the generalization good for FFNNs?

If one is training a basic FFNN (Feed-Forward Neural Network), one would apply regularizations like dropout, l1, l2 and gaussian noise, so that the model is robust and gives better results for unseen ...
-1
votes
1answer
45 views

How can I improve a model which is overly confident in the wrong thing?

I am currently estimating the certainty of a models estimation by running a neural network model with dropout multiple times and looking at the range of values. The results confuse me. I can group ...