Questions tagged [dropout]

Dropout is a technique to reduce overfitting during the training phase of a neural network.

Filter by
Sorted by
Tagged with
1
vote
0answers
22 views

Dropout onto pre-weighted vs onto pre-activated vector?

For any layer in my neural net, should I apply dropout onto an entering vector, or on the pre-activated vector? In other words: $$\vec q=W\cdot \vec x$$ $$\vec h = activate(drop(\vec q))$$ or: $$\...
3
votes
2answers
2k views

how to apply MC dropout to an LSTM network keras

I have a simple LSTM network developped using keras: ...
0
votes
1answer
29 views

Is generalizing a model, then removing the generalization good for FFNNs?

If one is training a basic FFNN (Feed-Forward Neural Network), one would apply regularizations like dropout, l1, l2 and gaussian noise, so that the model is robust and gives better results for unseen ...
1
vote
1answer
30 views

Using batchnorm and dropout simultaneously?

I am a bit confused about the relation between terms "Dropout" and "BatchNorm". As I understand, Dropout is regularization technique, which is using only during training. ...
0
votes
0answers
32 views

Parallel programming in Python

I have the next code that I am trying to run in parallel: ...
1
vote
2answers
351 views

Training a CNN on a large dataset

I am currently trying to build a CNN for around 100,000 images. There are 42 classes. I have used the default batch size of 32. This is how my model looks like: ...
0
votes
0answers
12 views

Prove Ridge regression duality of dropout

EDIT: Found one mistake, confused rows and columns in the step of summing over column operations. I'm trying to prove the Ridge regression duality of Dropout, as described in section 9.1 of this paper....
22
votes
5answers
10k views

Convolutional neural network overfitting. Dropout not helping

I am playing a little with convnets. Specifically, I am using the kaggle cats-vs-dogs dataset which consists on 25000 images labeled as either cat or dog (12500 each). I've managed to achieve around ...
0
votes
1answer
399 views

How is the validation set processed in PyTorch?

Say, one uses the MNIST dataset and splits the provided training data of size 60,000 into a training set (50,000) and a validation set (10,000). The provided test data of size 10,000 is used as the ...
2
votes
1answer
380 views

Monte Carlo Dropout as Uncertainty predection

I am pretty new to Python and this board so I am not sure, if I am at the right place for my question since it doesn't include any code. If not so, please give my a hint for a better way/place to ask. ...
0
votes
1answer
29 views

Lower training accuracy than testing accuracy (MLP/Dropout)

I am working on a problem of multi-class classification by MLP. I have set dropout to each middle layer. Now I observe the training accuracy is around 10% less than ...
11
votes
2answers
8k views

How exactly does DropOut work with convolutional layers?

Dropout (paper, explanation) sets the output of some neurons to zero. So for a MLP, you could have the following architecture for the Iris flower dataset: ...
0
votes
1answer
27 views

Can I apply Dropout In layers other than Fully Connected layers in CNN

I have read and seen that in CNN we apply DROPOUT layer between the FULLY CONNECTED layers to reduce overfitting. Can we also apply the dropout layer between the CONV layers and the POOL layers. I ...
14
votes
1answer
12k views

What is Monte Carlo dropout?

I understand how to use MC dropout from this answer, but I don't understand how MC dropout works, what its purpose is, and how it differs from normal dropout.
0
votes
0answers
11 views

Is the first Linear layer in Pytorch considered as the input layer?

For example: nn.Linear(100,50) You have 100 input features and 50 outputs. Would this be considered as the input layer or already as a kind of second layer? The problem: I don't want to apply dropout ...
1
vote
2answers
28 views

Structure of NN for input data with drop out

In financial markets, there is a simple problem of trading calendars varying across different countries. For example, Sweden observes Sweden National Day and Norway has Whit Monday. Typically, what ...
18
votes
1answer
15k views

When should one use L1, L2 regularization instead of dropout layer, given that both serve same purpose of reducing overfitting?

In Keras, there are 2 methods to reduce over-fitting. L1,L2 regularization or dropout layer. What are some situations to use L1,L2 regularization instead of dropout layer? What are some situations ...
1
vote
1answer
197 views

Should I set higher dropout prob if there are plenty of data?

I have some excessive amount of data for the size of NN I am able to teach in a reasonable time. If I feed all the data into the network it stops learning at some point and a resulting model shows ...
15
votes
5answers
5k views

Why does adding a dropout layer improve deep/machine learning performance, given that dropout suppresses some neurons from the model?

If removing some neurons results in a better performing model, why not use a simpler neural network with fewer layers and fewer neurons in the first place? Why build a bigger, more complicated model ...
1
vote
0answers
29 views

What is dropout in convolutional layers and how does that different from max-pooling-dropout?

When dropout is applied to fully connected layers some nodes will be randomly set to 0. It is unclear to me how dropout work with convolutional layers. If dropout is applied before the convolutions, ...
0
votes
0answers
89 views

Whats the difference between add.LSTM(num_hidden, droput=0.5) and add.Dropout(0.5) in Keras?

Could anyone please explain what is the difference between these two cases, specified in the title. I believe I am not the only one who is confused. I have read that it is preferrable to add Dropout ...
4
votes
2answers
395 views

Dropout in a CNN vs Dropout in a FCNN

In the PyTorch nn module there are 2 types of dropouts: A normal Dropout - During training, randomly zeroes some of the elements of the input tensor with ...
3
votes
1answer
1k views

LSTM: Converting to Bayesian Deep Neural Network

Starting from Yarin Gal's research paper on using Dropout as a Bayesian Approximation (https://arxiv.org/pdf/1506.02142.pdf), I am trying to apply this concept to my Sequence Prediction model. My ...
1
vote
0answers
39 views

GAN training the average of my train data

I have been training a GAN with 1D convolutional layers on sinus functions. However if I start varying my sinus (random amplitude for example), the model generates only the average of the random range....
4
votes
1answer
58 views

Bayesian regularization vs dropout for basic ann

Does it make sense conceptually to apply dropout to an artificial neutral network while also applying bayesian regularization? On one hand I would think that technically this should work just fine, ...
10
votes
2answers
7k views

Why should we use (or not) dropout on the input layer?

People generally avoid using dropout at the input layer itself. But wouldn't it be better to use it? Adding dropout (given that it's randomized it will probably end up acting like another regularizer)...
4
votes
1answer
9k views

How does dropout work during testing in neural network?

The below paragraph is picked from the textbook Hands-On Machine Learning with sci-kit learn & Tensorflow. I couldn't understand what the author is trying to ...
1
vote
1answer
254 views

Improving Accuracy of the Deep Learning Model

In my current project, I have only 647 rows (500 for training and 147 for testing) and I have applied the Keras Sequential model using the following code: ...
2
votes
0answers
24 views

Intuitive reasoning behind inverted dropout in neural networks

I'm going through the deeplearning.ai course on Coursera and am trying to understand the intuitive reasoning behind inverted dropout in neural networks. Based on the lecture, my understanding is as ...
1
vote
0answers
62 views

Is applying simultaneous K Fold Cross Validation and Drop out possible?

Well, it might seem ridiculous but I was just thinking whether it is possible to have these two methods simultaneously or not. I ran the code and faced an error, but in theory it doesn't seem ...
0
votes
1answer
80 views

Dropout dividing by compensation term = overshoots the result?

When applying dropout mask, why is it acceptable to divide the resulting state by the percentage of survived neurons? I understand that it's to prevent signal from dying out. But I've done the test, ...
3
votes
2answers
2k views

Regularization - Combine drop out with early stopping

I'm building a RNN (recurrent neural network) with LSTM cells. I'm using time series to perform anomaly detection. When training my RNN I'm using a dropout of 0.5 and I'm early stopping with a ...
2
votes
0answers
491 views
1
vote
1answer
83 views

Dropout implies stochastic descent?

The question is very simple, yet I can't find a quick confirmation on the web. It might seem obvious - by design, will Dropout always result in stochastic-looking gradient descent? (SGD) I've built ...
4
votes
1answer
757 views

Dropout in Deep Neural Networks

I was reading a paper published on Dropout. What I find difficulty in understanding that, In the training phase, a unit is present with a probability $p$ and not present with a probability $1-p$. In ...
13
votes
2answers
15k views

Dropout on which layers of LSTM?

Using a multi-layer LSTM with dropout, is it advisable to put dropout on all hidden layers as well as the output Dense layers? In Hinton's paper (which proposed ...
1
vote
1answer
196 views

How to interpret a drastic accuracy loss while training a neuronal net (CNN)?

How can one interpret a drastic accuracy loss after ~38 epochs? Maybe more dropout should be added to the CNN network? (x-axis shows the number of epochs)
1
vote
1answer
1k views

What is coadaptation of neurons in neural networks?

Looking for a bare minimum example (3 hidden units only maybe?) for what weights of a neural network with heavily coadapted weights would look like and showcase why they are bad. Also, how is ...
3
votes
1answer
807 views

Dropout on inputs instead on outputs = DropConnect?

Is dropping out parts of the Input vector better than dropping out parts of the Output vector? The latter literally makes this same neuron invisible to any further layers. On the contrary, ignoring ...
6
votes
1answer
4k views

Dropout vs weight decay

Dropout and weight decay are both regularization techniques. From my experience, dropout has been more widely used in the last few years. Are there scenarios where weight decay shines more than ...
3
votes
2answers
1k views

Convolutional layer dropout layer in keras

According to classical paper http://www.cs.toronto.edu/~rsalakhu/papers/srivastava14a.pdf dropout operation affects not only training step but also test step - we need to multiply all neuron output ...
2
votes
2answers
2k views

Dropout Decreases Test and Train Accuracy in one layer LSTM in Pytorch

I have a one layer lstm with pytorch on Mnist data. I know that for one layer lstm dropout option for lstm in pytorch does not operate. So, I have added a drop out at the beginning of second layer ...
2
votes
1answer
137 views

Dropout in other machine learning models

Dropout is a widely used technique in deep learning. Dropout was built for neural networks, but I wonder if other prediction models can use this idea as well as a regularizer. Do you know of any ...
3
votes
1answer
989 views

Probability of dropout growth

In the DNN literature, is there analysis or a term on a dropout ratio (oppositely-)proportional to the depth of a layer? By intuition, I'd like to dropout fewer neurons on the layers next to the ...
1
vote
0answers
297 views

Concrete Dropout for Recurrent Neural Networks (Keras)

I would like to use the Concrete Dropout Framework from GAL in application to recurrent neural networks. There is a great paper about it and the implementation can be found on the website (Thank you ...
2
votes
1answer
231 views

Do we need to increase training data size when increasing dropouts?

I am using a fully connected feed forward neural network built using keras for text classification. It consists of 3 hidden layer. I am planning to add a dropout layer after each hidden layer to ...
2
votes
1answer
63 views

How to think about prediction error that is not convex in hyperparameter, or over the course of training

Take the following case of a hyperparameter and prediction error: Imagine that the hyperparameter is a L2 penalty or a dropout rate -- something that we think that should have a single sweet spot -- ...
1
vote
1answer
100 views

Dropout without the averaging

The final step in dropout regularization is to multiply the weights by the dropout probability. This is motivated by analogy to bagging: averaging the weights of multiple nets. But it isn't truly that ...