3
$\begingroup$

Keras model looks like this

inp = Input(shape=(maxlen, ))
x = Embedding(max_features, embed_size, weights=[embedding_matrix], trainable=False)(inp)
x = SpatialDropout1D(dropout)(x)
x = Bidirectional(LSTM(num_filters, return_sequences=True))(x)
max_pool = GlobalMaxPooling1D()(x)
x = concatenate([x_h, max_pool])
outp = Dense(6, activation="sigmoid")(x)

According to the documentation the output shape = input shape i.e. (samples, timesteps, channels)

Questions

  1. What does SpatialDropout1D() really do to the output of Embedding()? I know the output of LSTM Embedding is of dimension (batch_size, steps, features).
  2. Does SpatialDropout1D() just randomly replace some values of word embedding of each word by 0?
  3. How is SpatialDropout1D() different from Dropout() in Keras?
$\endgroup$
  • $\begingroup$ Intuitively This version performs the same function as Dropout, however it drops entire 1D feature maps instead of individual elements.. $\endgroup$ – Aditya Sep 20 '18 at 4:00
  • $\begingroup$ What is a 1D feature maps in the context of Embedding()? $\endgroup$ – GeorgeOfTheRF Sep 20 '18 at 4:02
  • $\begingroup$ stackoverflow.com/questions/50393666/… $\endgroup$ – Aditya Sep 20 '18 at 4:37
  • $\begingroup$ Dropping the cols itself! Is what I can conclude.. need to play around them like visualisation of the embedding to confirm it! $\endgroup$ – Aditya Sep 20 '18 at 8:49
0
$\begingroup$

Basically, it removes all the pixel in a row from all channels. eg: take [[1,1,1], [2,4,5]], there are 3 points with values in 2 channels, by doing SpatialDropout1D it zeros an entire row ie all attributes of a point is set to 0; like [[1,1,0], [2,4,0]]

number of such choices would be 3C0 + 3C1+ 3C2 + 3C3 = 8

The intuition behind this is in many cases for an image the adjacent pixels are correlated, so hiding one of them is not helping much, rather hiding entire row, that's gotta make a difference. reference material

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.