Questions tagged [dropout]

Dropout is a technique to reduce overfitting during the training phase of a neural network.

Filter by
Sorted by
Tagged with
0
votes
0answers
23 views

Dropout noise shape when applying it on a series

I am training a neural network based on the Deep Sets framework (https://arxiv.org/abs/1703.06114, https://arxiv.org/abs/1810.05165). The basis of this approach is that one has a series of input ...
0
votes
0answers
43 views

Why are deep ensembles and monte carlo dropout never used simultanuously in uncertainty estimation

In papers on this topic, I have seen deep ensembles being compared to dropout monte carlo. I was wondering why they are never used simultanously, since adding dropout monte carlo to every member of an ...
1
vote
1answer
61 views

Multiply weights after using dropout in training - PyTorch

I have a Pytorch regression model as follows: ...
1
vote
1answer
39 views

Improving Accuracy of the Deep Learning Model

In my current project, I have only 647 rows (500 for training and 147 for testing) and I have applied the Keras Sequential model using the following code: ...
1
vote
0answers
124 views

Monte Carlo Dropout as Uncertainty predection

I am pretty new to Python and this board so I am not sure, if I am at the right place for my question since it doesn't include any code. If not so, please give my a hint for a better way/place to ask. ...
1
vote
0answers
13 views

Intuitive reasoning behind inverted dropout in neural networks

I'm going through the deeplearning.ai course on Coursera and am trying to understand the intuitive reasoning behind inverted dropout in neural networks. Based on the lecture, my understanding is as ...
0
votes
1answer
16 views

Is generalizing a model, then removing the generalization good for FFNNs?

If one is training a basic FFNN (Feed-Forward Neural Network), one would apply regularizations like dropout, l1, l2 and gaussian noise, so that the model is robust and gives better results for unseen ...
2
votes
2answers
42 views

Dropout in a CNN vs Dropout in a FCNN

In the PyTorch nn module there are 2 types of dropouts: A normal Dropout - During training, randomly zeroes some of the elements of the input tensor with ...
1
vote
0answers
36 views

Is applying simultaneous K Fold Cross Validation and Drop out possible?

Well, it might seem ridiculous but I was just thinking whether it is possible to have these two methods simultaneously or not. I ran the code and faced an error, but in theory it doesn't seem ...
0
votes
0answers
25 views

Dropout in theory VS Practical Implementation

Summarised from Deep Learning by Goodfellow Chapter 7, Page 262: When we use Bagging models we average over all the predictions over the different models, which is written as $\frac{1}{k} \sum_{i=1}...
1
vote
0answers
237 views

Measuring uncertainty in an LSTM network using dropout in keras/tensorflow

I've created a simple LSTM network for testing ...
2
votes
1answer
389 views

LSTM: Converting to Bayesian Deep Neural Network

Starting from Yarin Gal's research paper on using Dropout as a Bayesian Approximation (https://arxiv.org/pdf/1506.02142.pdf), I am trying to apply this concept to my Sequence Prediction model. My ...
2
votes
1answer
506 views

how to apply MC dropout to an LSTM network keras

I have a simple LSTM network developped using keras: ...
3
votes
1answer
1k views

How does dropout work during testing in neural network?

The below paragraph is picked from the textbook Hands-On Machine Learning with sci-kit learn & Tensorflow. I couldn't understand what the author is trying to ...
5
votes
1answer
3k views

What is Monte Carlo dropout?

I understand how to use MC dropout from this answer, but I don't understand how MC dropout works, what its purpose is, and how it differs from normal dropout.
3
votes
1answer
204 views

Dropout in Deep Neural Networks

I was reading a paper published on Dropout. What I find difficulty in understanding that, In the training phase, a unit is present with a probability $p$ and not present with a probability $1-p$. In ...
3
votes
1answer
1k views

What does SpatialDropout1D() do to output of Embedding() in Keras?

Keras model looks like this ...
5
votes
2answers
3k views

Why should we use (or not) dropout on the input layer?

People generally avoid using dropout at the input layer itself. But wouldn't it be better to use it? Adding dropout (given that it's randomized it will probably end up acting like another regularizer)...
10
votes
2answers
5k views

Dropout on which layers of LSTM?

Using a multi-layer LSTM with dropout, is it advisable to put dropout on all hidden layers as well as the output Dense layers? In Hinton's paper (which proposed ...
1
vote
1answer
133 views

How to interpret a drastic accuracy loss while training a neuronal net (CNN)?

How can one interpret a drastic accuracy loss after ~38 epochs? Maybe more dropout should be added to the CNN network? (x-axis shows the number of epochs)
5
votes
1answer
7k views

When should one use L1, L2 regularization instead of dropout layer, given that both serve same purpose of reducing overfitting?

In Keras, there are 2 methods to reduce over-fitting. L1,L2 regularization or dropout layer. https://keras.io/regularizers/ https://keras.io/layers/core/#dropout What are some situations to use L1,...
12
votes
4answers
3k views

Why does adding a dropout layer improve deep/machine learning performance, given that dropout suppresses some neurons from the model?

If removing some neurons results in a better performing model, why not use a simpler neural network with fewer layers and fewer neurons in the first place? Why build a bigger, more complicated model ...
1
vote
1answer
331 views

What is coadaptation of neurons in neural networks?

Looking for a bare minimum example (3 hidden units only maybe?) for what weights of a neural network with heavily coadapted weights would look like and showcase why they are bad. Also, how is ...
-1
votes
1answer
44 views

How can I improve a model which is overly confident in the wrong thing?

I am currently estimating the certainty of a models estimation by running a neural network model with dropout multiple times and looking at the range of values. The results confuse me. I can group ...
1
vote
0answers
19 views

Dropout onto pre-weighted vs onto pre-activated vector?

For any layer in my neural net, should I apply dropout onto an entering vector, or on the pre-activated vector? In other words: $$\vec q=W\cdot \vec x$$ $$\vec h = activate(drop(\vec q))$$ or: $$\...
3
votes
1answer
527 views

Dropout on inputs instead on outputs = DropConnect?

Is dropping out parts of the Input vector better than dropping out parts of the Output vector? The latter literally makes this same neuron invisible to any further layers. On the contrary, ignoring ...
2
votes
1answer
109 views

Dropout in other machine learning models

Dropout is a widely used technique in deep learning. Dropout was built for neural networks, but I wonder if other prediction models can use this idea as well as a regularizer. Do you know of any ...
6
votes
1answer
2k views

Dropout vs weight decay

Dropout and weight decay are both regularization techniques. From my experience, dropout has been more widely used in the last few years. Are there scenarios where weight decay shines more than ...
2
votes
2answers
1k views

Regularization - Combine drop out with early stopping

I'm building a RNN (recurrent neural network) with LSTM cells. I'm using time series to perform anomaly detection. When training my RNN I'm using a dropout of 0.5 and I'm early stopping with a ...
2
votes
2answers
1k views

Dropout Decreases Test and Train Accuracy in one layer LSTM in Pytorch

I have a one layer lstm with pytorch on Mnist data. I know that for one layer lstm dropout option for lstm in pytorch does not operate. So, I have added a drop out at the beginning of second layer ...
1
vote
0answers
242 views

Concrete Dropout for Recurrent Neural Networks (Keras)

I would like to use the Concrete Dropout Framework from GAL in application to recurrent neural networks. There is a great paper about it and the implementation can be found on the website (Thank you ...
2
votes
1answer
192 views

Do we need to increase training data size when increasing dropouts?

I am using a fully connected feed forward neural network built using keras for text classification. It consists of 3 hidden layer. I am planning to add a dropout layer after each hidden layer to ...
3
votes
1answer
829 views

Probability of dropout growth

In the DNN literature, is there analysis or a term on a dropout ratio (oppositely-)proportional to the depth of a layer? By intuition, I'd like to dropout fewer neurons on the layers next to the ...
1
vote
1answer
56 views

Dropout implies stochastic descent?

The question is very simple, yet I can't find a quick confirmation on the web. It might seem obvious - by design, will Dropout always result in stochastic-looking gradient descent? (SGD) I've built ...
0
votes
1answer
73 views

Dropout dividing by compensation term = overshoots the result?

When applying dropout mask, why is it acceptable to divide the resulting state by the percentage of survived neurons? I understand that it's to prevent signal from dying out. But I've done the test, ...
2
votes
1answer
53 views

How to think about prediction error that is not convex in hyperparameter, or over the course of training

Take the following case of a hyperparameter and prediction error: Imagine that the hyperparameter is a L2 penalty or a dropout rate -- something that we think that should have a single sweet spot -- ...
1
vote
1answer
74 views

Dropout without the averaging

The final step in dropout regularization is to multiply the weights by the dropout probability. This is motivated by analogy to bagging: averaging the weights of multiple nets. But it isn't truly that ...
1
vote
0answers
96 views

Why isn't Maxout used in the state of the art models?

I have just read the paper from Ian Goodfellow et al. titled "Maxout Networks". It seems that the Maxout activation should be quite powerful, as it can approximate any convex function, i.e. Relu, ...
0
votes
1answer
543 views

Loss of MSE always be 0 when keras for topic predict

my input is a 200 dims vector, which is generated by mean of the word2vector of all words of a article, my output is a 50 dims vector,which is generated by the LDA results of a article I want to use ...
16
votes
5answers
6k views

Convolutional neural network overfitting. Dropout not helping

I am playing a little with convnets. Specifically, I am using the kaggle cats-vs-dogs dataset which consists on 25000 images labeled as either cat or dog (12500 each). I've managed to achieve around ...
1
vote
1answer
151 views

Should I set higher dropout prob if there are plenty of data?

I have some excessive amount of data for the size of NN I am able to teach in a reasonable time. If I feed all the data into the network it stops learning at some point and a resulting model shows ...
1
vote
1answer
2k views

Dropout backpropagation implementation details

Just to summarize Understanding dropout and gradient descent and https://stats.stackexchange.com/questions/207481/dropout-backpropagation-implementation Suppose I need to implement inverted dropout ...
0
votes
1answer
768 views

Correct order of operations involved into Dropout

Suppose we have CNN with any hidden layer with activation followed by dropout layer. What is the correct precedence of activation and dropout operation if dropout implementation is inverted dropout ...
3
votes
2answers
1k views

Convolutional layer dropout layer in keras

According to classical paper http://www.cs.toronto.edu/~rsalakhu/papers/srivastava14a.pdf dropout operation affects not only training step but also test step - we need to multiply all neuron output ...
4
votes
1answer
2k views

Why does dropout ruin my accuracy in CNN?

I've build a CNN in Tensorflow with 2 conv layers, 1 pool layer and 2 FC layers. When I don't use dropout I get 98% accuracy on training dataset and 90% on test dataset. But, when I do use dropout, I ...
10
votes
1answer
6k views

How exactly does DropOut work with convolutional layers?

Dropout (paper, explanation) sets the output of some neurons to zero. So for a MLP, you could have the following architecture for the Iris flower dataset: ...
2
votes
1answer
86 views

Does dropout require multiple passes of the same data set, as a sort of ensemble method?

I'm a bit confused about dropout -- on one tutorial, it was described as basically an 'ensemble method' of sorts. This implies that you might need to create an ensemble of networks. Is this the case, ...
4
votes
1answer
162 views

In Neural Networks and deep neural networks what does label-dropout mean

If you take the following sentence from an article on deep neural networks to regularize the classifier layer by estimating the marginalized effect of label-dropout during training. What does ...
9
votes
2answers
147 views

Are there studies which examine dropout vs other regularizations?

Are there any papers published which show differences of the regularization methods for neural networks, preferably on different domains (or at least different datasets)? I am asking because I ...