6
$\begingroup$

Keras model looks like this

inp = Input(shape=(maxlen, ))
x = Embedding(max_features, embed_size, weights=[embedding_matrix], trainable=False)(inp)
x = SpatialDropout1D(dropout)(x)
x = Bidirectional(LSTM(num_filters, return_sequences=True))(x)
max_pool = GlobalMaxPooling1D()(x)
x = concatenate([x_h, max_pool])
outp = Dense(6, activation="sigmoid")(x)

According to the documentation the output shape = input shape i.e. (samples, timesteps, channels)

Questions

  1. What does SpatialDropout1D() really do to the output of Embedding()? I know the output of LSTM Embedding is of dimension (batch_size, steps, features).
  2. Does SpatialDropout1D() just randomly replace some values of word embedding of each word by 0?
  3. How is SpatialDropout1D() different from Dropout() in Keras?
Improve this question
$\endgroup$
4
  • $\begingroup$ Intuitively This version performs the same function as Dropout, however it drops entire 1D feature maps instead of individual elements.. $\endgroup$
    – Aditya
    Commented Sep 20, 2018 at 4:00
  • 1
    $\begingroup$ What is a 1D feature maps in the context of Embedding()? $\endgroup$
    – GeorgeOfTheRF
    Commented Sep 20, 2018 at 4:02
  • $\begingroup$ stackoverflow.com/questions/50393666/… $\endgroup$
    – Aditya
    Commented Sep 20, 2018 at 4:37
  • $\begingroup$ Dropping the cols itself! Is what I can conclude.. need to play around them like visualisation of the embedding to confirm it! $\endgroup$
    – Aditya
    Commented Sep 20, 2018 at 8:49

1 Answer 1

Reset to default
1
$\begingroup$

Basically, it removes all the pixel in a row from all channels. eg: take [[1,1,1], [2,4,5]], there are 3 points with values in 2 channels, by doing SpatialDropout1D it zeros an entire row ie all attributes of a point is set to 0; like [[1,1,0], [2,4,0]]

number of such choices would be 3C0 + 3C1+ 3C2 + 3C3 = 8

The intuition behind this is in many cases for an image the adjacent pixels are correlated, so hiding one of them is not helping much, rather hiding entire row, that's gotta make a difference. reference material

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.