0
$\begingroup$

I think I am not conceptually understanding "Dropout" in neural networks. I was under the assumption that a keep rate of 0.8 would set 20% of all the neurons to 0 for each training example.

In the code below, dropout is added as its own layer. What is happening here - Is it dropping neurons in the next layers? Or does mentioning dropout implement dropout in the entire model?

A clarification would be great. Thanks!

tf.keras.models.Sequential([
    tf.keras.layers.Flatten(input_shape=(28, 28), name='layers_flatten'),
    tf.keras.layers.Dense(512, activation='relu', name='layers_dense'),
    tf.keras.layers.Dropout(0.2, name='layers_dropout'),
    tf.keras.layers.Dense(10, activation='softmax', name='layers_dense_2')
  ])
$\endgroup$

1 Answer 1

1
$\begingroup$

you are just partially correct according to the TensorFlow implementation. Dropout set to 0.2 will set 20% of random input units to 0. Therefore, it does not affect neurons, just the previous layer's output and that's why it's not global and works only after the layer you put it in.

Also, it's noteworthy to mention that Dropout layer rescales remain non-zero outputs to violate the output's sum less. For instance, if I put [[1,0.5]] to a Dropout layer with a 0.5 rate, it would not output [[1,0]] or [[0,0.5]], but [[2,0]] or [[0,1]].

$\endgroup$
2
  • $\begingroup$ So it is analogous to a filtering layers that randomly allows only some percentage of information and sets the rest to 0? $\endgroup$
    – Yash Mali
    Dec 10, 2023 at 22:49
  • $\begingroup$ Yes, but it also rescales non-zero values. $\endgroup$ Dec 11, 2023 at 6:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.