I am training a neural network based on the Deep Sets framework (https://arxiv.org/abs/1703.06114, https://arxiv.org/abs/1810.05165). The basis of this approach is that one has a series of input objects, and applies a transformation on each of these objects to a semantic space (in the form of a dense neural network). The neural network applying the transformation is shared between all objects. In the semantic space the objects are summed and then used as inputs to a dense neural network.

I want to apply dropout on the neural network transforming the input objects, but am wondering what noise shape would be best to use. I am currently applying a dropout mask which can disable different nodes for each input object during training (though each object shares weights). Using Keras, the usage of such a dropout layer, and following dense layer looks as follows:

intermediate = layers.TimeDistributed( layers.Dropout( dropout_rate ) )( intermediate )
intermediate = layers.TimeDistributed( layers.Dense( number_of_nodes, activation = 'relu' ) )( intermediate )

Now I am wondering whether it would be advantageous to force dropout to apply the same noise for each of the input objects on a training sample (so the same nodes would be disabled for each object when doing the transformation to the semantic space), as follows:

intermediate = layers.SpatialDropout1D( dropout_rate )( intermediate )

i.e. using layers.SpatialDropout1D, which means the noise is the same for each timestep.

My understanding is that dropout essentially causes one to have a collection of thinned out neural networks which are each trained very rarely (or not at all), which are then effectively combined. Following this reasoning I wonder which of the above approaches should be expected to be the best. In the first case more of the weights in the neural network will have non-zero gradients on a single training sample, but they can not learn to conspire on a single object. But maybe a conspiracy across objects is still possible? In the second case one forces a single thinned network to be updated depending on each input object.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.