Questions tagged [dropout]

Dropout is a technique to reduce overfitting during the training phase of a neural network.

Filter by
Sorted by
Tagged with
0 votes
0 answers
40 views

Drop Out in Hyperparameter Optimisation

Is it correct to add dropout to each layer and that it is done as in the below example? class MyHyperModel(kt.HyperModel): def build_model(self, hp): ...
user avatar
  • 71
0 votes
0 answers
7 views

model loss is less but prediction is wrong

I have 100 samples having following data ...
user avatar
  • 1
0 votes
1 answer
36 views

What is the problem that causes overfitting in the code?

** ...
user avatar
1 vote
1 answer
46 views

Using batchnorm and dropout simultaneously?

I am a bit confused about the relation between terms "Dropout" and "BatchNorm". As I understand, Dropout is regularization technique, which is using only during training. ...
user avatar
  • 13
0 votes
1 answer
1k views

How is the validation set processed in PyTorch?

Say, one uses the MNIST dataset and splits the provided training data of size 60,000 into a training set (50,000) and a validation set (10,000). The provided test data of size 10,000 is used as the ...
user avatar
0 votes
1 answer
130 views

Lower training accuracy than testing accuracy (MLP/Dropout)

I am working on a problem of multi-class classification by MLP. I have set dropout to each middle layer. Now I observe the training accuracy is around 10% less than ...
user avatar
  • 103
0 votes
1 answer
37 views

Can I apply Dropout In layers other than Fully Connected layers in CNN

I have read and seen that in CNN we apply DROPOUT layer between the FULLY CONNECTED layers to reduce overfitting. Can we also apply the dropout layer between the CONV layers and the POOL layers. I ...
user avatar
  • 669
1 vote
2 answers
30 views

Structure of NN for input data with drop out

In financial markets, there is a simple problem of trading calendars varying across different countries. For example, Sweden observes Sweden National Day and Norway has Whit Monday. Typically, what ...
user avatar
  • 171
1 vote
2 answers
615 views

Training a CNN on a large dataset

I am currently trying to build a CNN for around 100,000 images. There are 42 classes. I have used the default batch size of 32. This is how my model looks like: ...
user avatar
  • 11
1 vote
0 answers
53 views

What is dropout in convolutional layers and how does that different from max-pooling-dropout?

When dropout is applied to fully connected layers some nodes will be randomly set to 0. It is unclear to me how dropout work with convolutional layers. If dropout is applied before the convolutions, ...
user avatar
0 votes
0 answers
137 views

Whats the difference between add.LSTM(num_hidden, droput=0.5) and add.Dropout(0.5) in Keras?

Could anyone please explain what is the difference between these two cases, specified in the title. I believe I am not the only one who is confused. I have read that it is preferrable to add Dropout ...
user avatar
  • 19
1 vote
0 answers
43 views

GAN training the average of my train data

I have been training a GAN with 1D convolutional layers on sinus functions. However if I start varying my sinus (random amplitude for example), the model generates only the average of the random range....
user avatar
4 votes
1 answer
84 views

Bayesian regularization vs dropout for basic ann

Does it make sense conceptually to apply dropout to an artificial neutral network while also applying bayesian regularization? On one hand I would think that technically this should work just fine, ...
user avatar
  • 143
1 vote
1 answer
440 views

Multiply weights after using dropout in training - PyTorch

I have a Pytorch regression model as follows: ...
user avatar
1 vote
1 answer
330 views

Improving Accuracy of the Deep Learning Model

In my current project, I have only 647 rows (500 for training and 147 for testing) and I have applied the Keras Sequential model using the following code: ...
user avatar
2 votes
1 answer
478 views

Monte Carlo Dropout as Uncertainty predection

I am pretty new to Python and this board so I am not sure, if I am at the right place for my question since it doesn't include any code. If not so, please give my a hint for a better way/place to ask. ...
user avatar
2 votes
0 answers
34 views

Intuitive reasoning behind inverted dropout in neural networks

I'm going through the deeplearning.ai course on Coursera and am trying to understand the intuitive reasoning behind inverted dropout in neural networks. Based on the lecture, my understanding is as ...
user avatar
0 votes
1 answer
36 views

Is generalizing a model, then removing the generalization good for FFNNs?

If one is training a basic FFNN (Feed-Forward Neural Network), one would apply regularizations like dropout, l1, l2 and gaussian noise, so that the model is robust and gives better results for unseen ...
user avatar
4 votes
2 answers
687 views

Dropout in a CNN vs Dropout in a FCNN

In the PyTorch nn module there are 2 types of dropouts: A normal Dropout - During training, randomly zeroes some of the elements of the input tensor with ...
user avatar
  • 753
1 vote
0 answers
109 views

Is applying simultaneous K Fold Cross Validation and Drop out possible?

Well, it might seem ridiculous but I was just thinking whether it is possible to have these two methods simultaneously or not. I ran the code and faced an error, but in theory it doesn't seem ...
user avatar
2 votes
0 answers
550 views

Measuring uncertainty in an LSTM network using dropout in keras/tensorflow

I've created a simple LSTM network for testing ...
user avatar
  • 21
3 votes
1 answer
1k views

LSTM: Converting to Bayesian Deep Neural Network

Starting from Yarin Gal's research paper on using Dropout as a Bayesian Approximation (https://arxiv.org/pdf/1506.02142.pdf), I am trying to apply this concept to my Sequence Prediction model. My ...
user avatar
  • 31
4 votes
2 answers
2k views

how to apply MC dropout to an LSTM network keras

I have a simple LSTM network developped using keras: ...
user avatar
  • 41
8 votes
1 answer
15k views

How does dropout work during testing in neural network?

The below paragraph is picked from the textbook Hands-On Machine Learning with sci-kit learn & Tensorflow. I couldn't understand what the author is trying to ...
user avatar
  • 437
19 votes
1 answer
15k views

What is Monte Carlo dropout?

I understand how to use MC dropout from this answer, but I don't understand how MC dropout works, what its purpose is, and how it differs from normal dropout.
user avatar
  • 530
3 votes
1 answer
1k views

Dropout in Deep Neural Networks

I was reading a paper published on Dropout. What I find difficulty in understanding that, In the training phase, a unit is present with a probability $p$ and not present with a probability $1-p$. In ...
user avatar
6 votes
1 answer
3k views

What does SpatialDropout1D() do to output of Embedding() in Keras?

Keras model looks like this ...
user avatar
11 votes
2 answers
9k views

Why should we use (or not) dropout on the input layer?

People generally avoid using dropout at the input layer itself. But wouldn't it be better to use it? Adding dropout (given that it's randomized it will probably end up acting like another regularizer)...
user avatar
  • 2,400
14 votes
2 answers
22k views

Dropout on which layers of LSTM?

Using a multi-layer LSTM with dropout, is it advisable to put dropout on all hidden layers as well as the output Dense layers? In Hinton's paper (which proposed ...
user avatar
  • 730
1 vote
1 answer
213 views

How to interpret a drastic accuracy loss while training a neuronal net (CNN)?

How can one interpret a drastic accuracy loss after ~38 epochs? Maybe more dropout should be added to the CNN network? (x-axis shows the number of epochs)
user avatar
24 votes
2 answers
19k views

When should one use L1, L2 regularization instead of dropout layer, given that both serve same purpose of reducing overfitting?

In Keras, there are 2 methods to reduce over-fitting. L1,L2 regularization or dropout layer. What are some situations to use L1,L2 regularization instead of dropout layer? What are some situations ...
user avatar
  • 1,195
15 votes
5 answers
6k views

Why does adding a dropout layer improve deep/machine learning performance, given that dropout suppresses some neurons from the model?

If removing some neurons results in a better performing model, why not use a simpler neural network with fewer layers and fewer neurons in the first place? Why build a bigger, more complicated model ...
user avatar
  • 1,195
2 votes
1 answer
1k views

What is coadaptation of neurons in neural networks?

Looking for a bare minimum example (3 hidden units only maybe?) for what weights of a neural network with heavily coadapted weights would look like and showcase why they are bad. Also, how is ...
user avatar
1 vote
0 answers
27 views

Dropout onto pre-weighted vs onto pre-activated vector?

For any layer in my neural net, should I apply dropout onto an entering vector, or on the pre-activated vector? In other words: $$\vec q=W\cdot \vec x$$ $$\vec h = activate(drop(\vec q))$$ or: $$\...
user avatar
  • 2,586
3 votes
1 answer
928 views

Dropout on inputs instead on outputs = DropConnect?

Is dropping out parts of the Input vector better than dropping out parts of the Output vector? The latter literally makes this same neuron invisible to any further layers. On the contrary, ignoring ...
user avatar
  • 2,586
2 votes
1 answer
153 views

Dropout in other machine learning models

Dropout is a widely used technique in deep learning. Dropout was built for neural networks, but I wonder if other prediction models can use this idea as well as a regularizer. Do you know of any ...
user avatar
  • 5,664
6 votes
1 answer
4k views

Dropout vs weight decay

Dropout and weight decay are both regularization techniques. From my experience, dropout has been more widely used in the last few years. Are there scenarios where weight decay shines more than ...
user avatar
  • 5,664
4 votes
2 answers
3k views

Regularization - Combine drop out with early stopping

I'm building a RNN (recurrent neural network) with LSTM cells. I'm using time series to perform anomaly detection. When training my RNN I'm using a dropout of 0.5 and I'm early stopping with a ...
user avatar
2 votes
2 answers
3k views

Dropout Decreases Test and Train Accuracy in one layer LSTM in Pytorch

I have a one layer lstm with pytorch on Mnist data. I know that for one layer lstm dropout option for lstm in pytorch does not operate. So, I have added a drop out at the beginning of second layer ...
user avatar
1 vote
0 answers
321 views

Concrete Dropout for Recurrent Neural Networks (Keras)

I would like to use the Concrete Dropout Framework from GAL in application to recurrent neural networks. There is a great paper about it and the implementation can be found on the website (Thank you ...
user avatar
  • 259
2 votes
1 answer
242 views

Do we need to increase training data size when increasing dropouts?

I am using a fully connected feed forward neural network built using keras for text classification. It consists of 3 hidden layer. I am planning to add a dropout layer after each hidden layer to ...
user avatar
3 votes
1 answer
1k views

Probability of dropout growth

In the DNN literature, is there analysis or a term on a dropout ratio (oppositely-)proportional to the depth of a layer? By intuition, I'd like to dropout fewer neurons on the layers next to the ...
user avatar
  • 363
1 vote
1 answer
108 views

Dropout implies stochastic descent?

The question is very simple, yet I can't find a quick confirmation on the web. It might seem obvious - by design, will Dropout always result in stochastic-looking gradient descent? (SGD) I've built ...
user avatar
  • 2,586
0 votes
1 answer
84 views

Dropout dividing by compensation term = overshoots the result?

When applying dropout mask, why is it acceptable to divide the resulting state by the percentage of survived neurons? I understand that it's to prevent signal from dying out. But I've done the test, ...
user avatar
  • 2,586
2 votes
1 answer
68 views

How to think about prediction error that is not convex in hyperparameter, or over the course of training

Take the following case of a hyperparameter and prediction error: Imagine that the hyperparameter is a L2 penalty or a dropout rate -- something that we think that should have a single sweet spot -- ...
user avatar
1 vote
1 answer
124 views

Dropout without the averaging

The final step in dropout regularization is to multiply the weights by the dropout probability. This is motivated by analogy to bagging: averaging the weights of multiple nets. But it isn't truly that ...
user avatar
1 vote
0 answers
131 views

Why isn't Maxout used in the state of the art models?

I have just read the paper from Ian Goodfellow et al. titled "Maxout Networks". It seems that the Maxout activation should be quite powerful, as it can approximate any convex function, i.e. Relu, ...
user avatar
  • 111
0 votes
1 answer
723 views

Loss of MSE always be 0 when keras for topic predict

my input is a 200 dims vector, which is generated by mean of the word2vector of all words of a article, my output is a 50 dims vector,which is generated by the LDA results of a article I want to use ...
user avatar
23 votes
5 answers
10k views

Convolutional neural network overfitting. Dropout not helping

I am playing a little with convnets. Specifically, I am using the kaggle cats-vs-dogs dataset which consists on 25000 images labeled as either cat or dog (12500 each). I've managed to achieve around ...
user avatar
1 vote
1 answer
214 views

Should I set higher dropout prob if there are plenty of data?

I have some excessive amount of data for the size of NN I am able to teach in a reasonable time. If I feed all the data into the network it stops learning at some point and a resulting model shows ...
user avatar