Questions tagged [dropout]

Dropout is a technique to reduce overfitting during the training phase of a neural network.

15 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
3
votes
1answer
1k views

What does SpatialDropout1D() do to output of Embedding() in Keras?

Keras model looks like this ...
2
votes
2answers
43 views

Dropout in a CNN vs Dropout in a FCNN

In the PyTorch nn module there are 2 types of dropouts: A normal Dropout - During training, randomly zeroes some of the elements of the input tensor with ...
2
votes
1answer
401 views

LSTM: Converting to Bayesian Deep Neural Network

Starting from Yarin Gal's research paper on using Dropout as a Bayesian Approximation (https://arxiv.org/pdf/1506.02142.pdf), I am trying to apply this concept to my Sequence Prediction model. My ...
1
vote
0answers
126 views

Monte Carlo Dropout as Uncertainty predection

I am pretty new to Python and this board so I am not sure, if I am at the right place for my question since it doesn't include any code. If not so, please give my a hint for a better way/place to ask. ...
1
vote
0answers
13 views

Intuitive reasoning behind inverted dropout in neural networks

I'm going through the deeplearning.ai course on Coursera and am trying to understand the intuitive reasoning behind inverted dropout in neural networks. Based on the lecture, my understanding is as ...
1
vote
0answers
36 views

Is applying simultaneous K Fold Cross Validation and Drop out possible?

Well, it might seem ridiculous but I was just thinking whether it is possible to have these two methods simultaneously or not. I ran the code and faced an error, but in theory it doesn't seem ...
1
vote
0answers
237 views

Measuring uncertainty in an LSTM network using dropout in keras/tensorflow

I've created a simple LSTM network for testing ...
1
vote
0answers
19 views

Dropout onto pre-weighted vs onto pre-activated vector?

For any layer in my neural net, should I apply dropout onto an entering vector, or on the pre-activated vector? In other words: $$\vec q=W\cdot \vec x$$ $$\vec h = activate(drop(\vec q))$$ or: $$\...
1
vote
0answers
243 views

Concrete Dropout for Recurrent Neural Networks (Keras)

I would like to use the Concrete Dropout Framework from GAL in application to recurrent neural networks. There is a great paper about it and the implementation can be found on the website (Thank you ...
1
vote
0answers
96 views

Why isn't Maxout used in the state of the art models?

I have just read the paper from Ian Goodfellow et al. titled "Maxout Networks". It seems that the Maxout activation should be quite powerful, as it can approximate any convex function, i.e. Relu, ...
0
votes
0answers
24 views

Dropout noise shape when applying it on a series

I am training a neural network based on the Deep Sets framework (https://arxiv.org/abs/1703.06114, https://arxiv.org/abs/1810.05165). The basis of this approach is that one has a series of input ...
0
votes
0answers
45 views

Why are deep ensembles and monte carlo dropout never used simultanuously in uncertainty estimation

In papers on this topic, I have seen deep ensembles being compared to dropout monte carlo. I was wondering why they are never used simultanously, since adding dropout monte carlo to every member of an ...
0
votes
1answer
16 views

Is generalizing a model, then removing the generalization good for FFNNs?

If one is training a basic FFNN (Feed-Forward Neural Network), one would apply regularizations like dropout, l1, l2 and gaussian noise, so that the model is robust and gives better results for unseen ...
0
votes
0answers
25 views

Dropout in theory VS Practical Implementation

Summarised from Deep Learning by Goodfellow Chapter 7, Page 262: When we use Bagging models we average over all the predictions over the different models, which is written as $\frac{1}{k} \sum_{i=1}...
-1
votes
1answer
44 views

How can I improve a model which is overly confident in the wrong thing?

I am currently estimating the certainty of a models estimation by running a neural network model with dropout multiple times and looking at the range of values. The results confuse me. I can group ...