2
$\begingroup$

CNNs are used in NLP for various tasks. But I cannot find a clear understanding of why do we only use 1d filters in these networks?

$\endgroup$
2
$\begingroup$

Text is a 1D sequence, but is typically treated as a sequence of embedding vectors. So yes it is in some sense 2D input. But the embedding dimension doesn't really have any spatial meaning; adjacent dimensions aren't any more related than any others. There is no invariance across the embedding dimension either; the same values in one part of the embedding don't mean the same thing. So the assumptions a 2D convolution don't make sense for this type of input.

$\endgroup$
  • $\begingroup$ by 1D, you mean using filter (n * full embedding length) or (n * 1)? My CNN performs better when I use filter of 3*1 than 3*100 (the size of the word embedding). Do you know any papers where this issues is talked through because I feel like everybody just uses the full length of the embedding as the second dimension, but I did not see any good explanation. What if we get some information out of convolving each dimension of the embedding separately (using smaller filter) because there might be some relationship between the values of the embedding of more words in the same dimension? $\endgroup$ – Jakub Oct 10 '20 at 19:00
  • $\begingroup$ Yeah I suppose I should clarify. OK, if your input is n characters x m embedding dims, then a filter of say (3 x m) makes sense and that is "2D". (3 x 1) also sort of makes sense - you learn about how each embedding dim changes over a window of time, but I doubt that's as useful. I don't see that anything else makes much sense as it would be pretending that adjacent dimensions have more relationship, and that it's the same relationship across the dimensions, but that isn't true. $\endgroup$ – Sean Owen Oct 11 '20 at 13:50
  • $\begingroup$ Thanks. Could you refer me to any papers which discuss this in any way? I didn't find anything, but taking it "for granted" does not sound right. $\endgroup$ – Jakub Oct 13 '20 at 18:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.