I am training a neural network based on the Deep Sets framework (https://arxiv.org/abs/1703.06114, https://arxiv.org/abs/1810.05165). The basis of this approach is that one has a series of input objects, and applies a transformation on each of these objects to a semantic space (in the form of a dense neural network). The neural network applying the transformation is shared between all objects. In the semantic space the objects are summed and then used as inputs to a dense neural network.
I want to apply dropout on the neural network transforming the input objects, but am wondering what noise shape would be best to use. I am currently applying a dropout mask which can disable different nodes for each input object during training (though each object shares weights). Using Keras, the usage of such a dropout layer, and following dense layer looks as follows:
intermediate = layers.TimeDistributed( layers.Dropout( dropout_rate ) )( intermediate ) intermediate = layers.TimeDistributed( layers.Dense( number_of_nodes, activation = 'relu' ) )( intermediate )
Now I am wondering whether it would be advantageous to force dropout to apply the same noise for each of the input objects on a training sample (so the same nodes would be disabled for each object when doing the transformation to the semantic space), as follows:
intermediate = layers.SpatialDropout1D( dropout_rate )( intermediate )
i.e. using layers.SpatialDropout1D, which means the noise is the same for each timestep.
My understanding is that dropout essentially causes one to have a collection of thinned out neural networks which are each trained very rarely (or not at all), which are then effectively combined. Following this reasoning I wonder which of the above approaches should be expected to be the best. In the first case more of the weights in the neural network will have non-zero gradients on a single training sample, but they can not learn to conspire on a single object. But maybe a conspiracy across objects is still possible? In the second case one forces a single thinned network to be updated depending on each input object.