Questions tagged [dropout]

Dropout is a technique to reduce overfitting during the training phase of a neural network.

Filter by
Sorted by
Tagged with
1
vote
1answer
24 views

Using batchnorm and dropout simultaneously?

I am a bit confused about the relation between terms "Dropout" and "BatchNorm". As I understand, Dropout is regularization technique, which is using only during training. ...
0
votes
0answers
30 views

Parallel programming in Python

I have the next code that I am trying to run in parallel: ...
0
votes
0answers
10 views

Prove Ridge regression duality of dropout

EDIT: Found one mistake, confused rows and columns in the step of summing over column operations. I'm trying to prove the Ridge regression duality of Dropout, as described in section 9.1 of this paper....
0
votes
1answer
274 views

How is the validation set processed in PyTorch?

Say, one uses the MNIST dataset and splits the provided training data of size 60,000 into a training set (50,000) and a validation set (10,000). The provided test data of size 10,000 is used as the ...
0
votes
1answer
26 views

Lower training accuracy than testing accuracy (MLP/Dropout)

I am working on a problem of multi-class classification by MLP. I have set dropout to each middle layer. Now I observe the training accuracy is around 10% less than ...
0
votes
1answer
26 views

Can I apply Dropout In layers other than Fully Connected layers in CNN

I have read and seen that in CNN we apply DROPOUT layer between the FULLY CONNECTED layers to reduce overfitting. Can we also apply the dropout layer between the CONV layers and the POOL layers. I ...
0
votes
0answers
11 views

Is the first Linear layer in Pytorch considered as the input layer?

For example: nn.Linear(100,50) You have 100 input features and 50 outputs. Would this be considered as the input layer or already as a kind of second layer? The problem: I don't want to apply dropout ...
1
vote
2answers
28 views

Structure of NN for input data with drop out

In financial markets, there is a simple problem of trading calendars varying across different countries. For example, Sweden observes Sweden National Day and Norway has Whit Monday. Typically, what ...
1
vote
2answers
331 views

Training a CNN on a large dataset

I am currently trying to build a CNN for around 100,000 images. There are 42 classes. I have used the default batch size of 32. This is how my model looks like: ...
1
vote
0answers
25 views

What is dropout in convolutional layers and how does that different from max-pooling-dropout?

When dropout is applied to fully connected layers some nodes will be randomly set to 0. It is unclear to me how dropout work with convolutional layers. If dropout is applied before the convolutions, ...
0
votes
0answers
79 views

Whats the difference between add.LSTM(num_hidden, droput=0.5) and add.Dropout(0.5) in Keras?

Could anyone please explain what is the difference between these two cases, specified in the title. I believe I am not the only one who is confused. I have read that it is preferrable to add Dropout ...
1
vote
0answers
38 views

GAN training the average of my train data

I have been training a GAN with 1D convolutional layers on sinus functions. However if I start varying my sinus (random amplitude for example), the model generates only the average of the random range....
4
votes
1answer
56 views

Bayesian regularization vs dropout for basic ann

Does it make sense conceptually to apply dropout to an artificial neutral network while also applying bayesian regularization? On one hand I would think that technically this should work just fine, ...
1
vote
1answer
280 views

Multiply weights after using dropout in training - PyTorch

I have a Pytorch regression model as follows: ...
1
vote
1answer
242 views

Improving Accuracy of the Deep Learning Model

In my current project, I have only 647 rows (500 for training and 147 for testing) and I have applied the Keras Sequential model using the following code: ...
2
votes
1answer
363 views

Monte Carlo Dropout as Uncertainty predection

I am pretty new to Python and this board so I am not sure, if I am at the right place for my question since it doesn't include any code. If not so, please give my a hint for a better way/place to ask. ...
2
votes
0answers
24 views

Intuitive reasoning behind inverted dropout in neural networks

I'm going through the deeplearning.ai course on Coursera and am trying to understand the intuitive reasoning behind inverted dropout in neural networks. Based on the lecture, my understanding is as ...
0
votes
1answer
27 views

Is generalizing a model, then removing the generalization good for FFNNs?

If one is training a basic FFNN (Feed-Forward Neural Network), one would apply regularizations like dropout, l1, l2 and gaussian noise, so that the model is robust and gives better results for unseen ...
4
votes
2answers
363 views

Dropout in a CNN vs Dropout in a FCNN

In the PyTorch nn module there are 2 types of dropouts: A normal Dropout - During training, randomly zeroes some of the elements of the input tensor with ...
1
vote
0answers
55 views

Is applying simultaneous K Fold Cross Validation and Drop out possible?

Well, it might seem ridiculous but I was just thinking whether it is possible to have these two methods simultaneously or not. I ran the code and faced an error, but in theory it doesn't seem ...
2
votes
0answers
481 views

Measuring uncertainty in an LSTM network using dropout in keras/tensorflow

I've created a simple LSTM network for testing ...
3
votes
1answer
994 views

LSTM: Converting to Bayesian Deep Neural Network

Starting from Yarin Gal's research paper on using Dropout as a Bayesian Approximation (https://arxiv.org/pdf/1506.02142.pdf), I am trying to apply this concept to my Sequence Prediction model. My ...
2
votes
1answer
2k views

how to apply MC dropout to an LSTM network keras

I have a simple LSTM network developped using keras: ...
4
votes
1answer
9k views

How does dropout work during testing in neural network?

The below paragraph is picked from the textbook Hands-On Machine Learning with sci-kit learn & Tensorflow. I couldn't understand what the author is trying to ...
14
votes
1answer
11k views

What is Monte Carlo dropout?

I understand how to use MC dropout from this answer, but I don't understand how MC dropout works, what its purpose is, and how it differs from normal dropout.
3
votes
1answer
725 views

Dropout in Deep Neural Networks

I was reading a paper published on Dropout. What I find difficulty in understanding that, In the training phase, a unit is present with a probability $p$ and not present with a probability $1-p$. In ...
6
votes
1answer
3k views

What does SpatialDropout1D() do to output of Embedding() in Keras?

Keras model looks like this ...
9
votes
2answers
7k views

Why should we use (or not) dropout on the input layer?

People generally avoid using dropout at the input layer itself. But wouldn't it be better to use it? Adding dropout (given that it's randomized it will probably end up acting like another regularizer)...
13
votes
2answers
15k views

Dropout on which layers of LSTM?

Using a multi-layer LSTM with dropout, is it advisable to put dropout on all hidden layers as well as the output Dense layers? In Hinton's paper (which proposed ...
1
vote
1answer
195 views

How to interpret a drastic accuracy loss while training a neuronal net (CNN)?

How can one interpret a drastic accuracy loss after ~38 epochs? Maybe more dropout should be added to the CNN network? (x-axis shows the number of epochs)
18
votes
1answer
15k views

When should one use L1, L2 regularization instead of dropout layer, given that both serve same purpose of reducing overfitting?

In Keras, there are 2 methods to reduce over-fitting. L1,L2 regularization or dropout layer. What are some situations to use L1,L2 regularization instead of dropout layer? What are some situations ...
15
votes
5answers
5k views

Why does adding a dropout layer improve deep/machine learning performance, given that dropout suppresses some neurons from the model?

If removing some neurons results in a better performing model, why not use a simpler neural network with fewer layers and fewer neurons in the first place? Why build a bigger, more complicated model ...
1
vote
1answer
984 views

What is coadaptation of neurons in neural networks?

Looking for a bare minimum example (3 hidden units only maybe?) for what weights of a neural network with heavily coadapted weights would look like and showcase why they are bad. Also, how is ...
1
vote
0answers
20 views

Dropout onto pre-weighted vs onto pre-activated vector?

For any layer in my neural net, should I apply dropout onto an entering vector, or on the pre-activated vector? In other words: $$\vec q=W\cdot \vec x$$ $$\vec h = activate(drop(\vec q))$$ or: $$\...
3
votes
1answer
799 views

Dropout on inputs instead on outputs = DropConnect?

Is dropping out parts of the Input vector better than dropping out parts of the Output vector? The latter literally makes this same neuron invisible to any further layers. On the contrary, ignoring ...
2
votes
1answer
136 views

Dropout in other machine learning models

Dropout is a widely used technique in deep learning. Dropout was built for neural networks, but I wonder if other prediction models can use this idea as well as a regularizer. Do you know of any ...
6
votes
1answer
3k views

Dropout vs weight decay

Dropout and weight decay are both regularization techniques. From my experience, dropout has been more widely used in the last few years. Are there scenarios where weight decay shines more than ...
3
votes
2answers
2k views

Regularization - Combine drop out with early stopping

I'm building a RNN (recurrent neural network) with LSTM cells. I'm using time series to perform anomaly detection. When training my RNN I'm using a dropout of 0.5 and I'm early stopping with a ...
2
votes
2answers
2k views

Dropout Decreases Test and Train Accuracy in one layer LSTM in Pytorch

I have a one layer lstm with pytorch on Mnist data. I know that for one layer lstm dropout option for lstm in pytorch does not operate. So, I have added a drop out at the beginning of second layer ...
1
vote
0answers
296 views

Concrete Dropout for Recurrent Neural Networks (Keras)

I would like to use the Concrete Dropout Framework from GAL in application to recurrent neural networks. There is a great paper about it and the implementation can be found on the website (Thank you ...
2
votes
1answer
229 views

Do we need to increase training data size when increasing dropouts?

I am using a fully connected feed forward neural network built using keras for text classification. It consists of 3 hidden layer. I am planning to add a dropout layer after each hidden layer to ...
3
votes
1answer
987 views

Probability of dropout growth

In the DNN literature, is there analysis or a term on a dropout ratio (oppositely-)proportional to the depth of a layer? By intuition, I'd like to dropout fewer neurons on the layers next to the ...
1
vote
1answer
80 views

Dropout implies stochastic descent?

The question is very simple, yet I can't find a quick confirmation on the web. It might seem obvious - by design, will Dropout always result in stochastic-looking gradient descent? (SGD) I've built ...
0
votes
1answer
80 views

Dropout dividing by compensation term = overshoots the result?

When applying dropout mask, why is it acceptable to divide the resulting state by the percentage of survived neurons? I understand that it's to prevent signal from dying out. But I've done the test, ...
2
votes
1answer
62 views

How to think about prediction error that is not convex in hyperparameter, or over the course of training

Take the following case of a hyperparameter and prediction error: Imagine that the hyperparameter is a L2 penalty or a dropout rate -- something that we think that should have a single sweet spot -- ...
1
vote
1answer
99 views

Dropout without the averaging

The final step in dropout regularization is to multiply the weights by the dropout probability. This is motivated by analogy to bagging: averaging the weights of multiple nets. But it isn't truly that ...
1
vote
0answers
116 views

Why isn't Maxout used in the state of the art models?

I have just read the paper from Ian Goodfellow et al. titled "Maxout Networks". It seems that the Maxout activation should be quite powerful, as it can approximate any convex function, i.e. Relu, ...
0
votes
1answer
658 views

Loss of MSE always be 0 when keras for topic predict

my input is a 200 dims vector, which is generated by mean of the word2vector of all words of a article, my output is a 50 dims vector,which is generated by the LDA results of a article I want to use ...
21
votes
5answers
10k views

Convolutional neural network overfitting. Dropout not helping

I am playing a little with convnets. Specifically, I am using the kaggle cats-vs-dogs dataset which consists on 25000 images labeled as either cat or dog (12500 each). I've managed to achieve around ...
1
vote
1answer
195 views

Should I set higher dropout prob if there are plenty of data?

I have some excessive amount of data for the size of NN I am able to teach in a reasonable time. If I feed all the data into the network it stops learning at some point and a resulting model shows ...