0
$\begingroup$

I am doing some study on https://www.kaggle.com/anshulrai/cudnnlstm-implementation-93-7-accuracy

I understand we need LSTM to capture the sequence of words in the sentience, but I am not quite understand what does Conv1d do in the model architecture? Could someone please share some intuitive explanation?

Thanks a lot!

Model: "model_1"
__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
input_1 (InputLayer)            (None, 100)          0                                            
__________________________________________________________________________________________________
embedding_1 (Embedding)         (None, 100, 100)     2000000     input_1[0][0]                    
__________________________________________________________________________________________________
dropout_1 (Dropout)             (None, 100, 100)     0           embedding_1[0][0]                
__________________________________________________________________________________________________
conv1d_1 (Conv1D)               (None, 98, 200)      60200       dropout_1[0][0]                  
__________________________________________________________________________________________________
conv1d_2 (Conv1D)               (None, 96, 200)      120200      conv1d_1[0][0]                   
__________________________________________________________________________________________________
conv1d_3 (Conv1D)               (None, 94, 256)      153856      conv1d_2[0][0]                   
__________________________________________________________________________________________________
conv1d_4 (Conv1D)               (None, 92, 256)      196864      conv1d_3[0][0]                   
__________________________________________________________________________________________________
conv1d_5 (Conv1D)               (None, 45, 512)      393728      conv1d_4[0][0]                   
__________________________________________________________________________________________________
cu_dnnlstm_1 (CuDNNLSTM)        (None, 512)          2101248     conv1d_5[0][0]                   
__________________________________________________________________________________________________
cu_dnnlstm_2 (CuDNNLSTM)        (None, 512)          2101248     conv1d_5[0][0]                   
__________________________________________________________________________________________________
concatenate_1 (Concatenate)     (None, 1024)         0           cu_dnnlstm_1[0][0]               
                                                                 cu_dnnlstm_2[0][0]               
__________________________________________________________________________________________________
dropout_2 (Dropout)             (None, 1024)         0           concatenate_1[0][0]              
__________________________________________________________________________________________________
dense_1 (Dense)                 (None, 64)           65600       dropout_2[0][0]                  
__________________________________________________________________________________________________
dropout_3 (Dropout)             (None, 64)           0           dense_1[0][0]                    
__________________________________________________________________________________________________
dense_2 (Dense)                 (None, 1)            65          dropout_3[0][0]                  
==================================================================================================
Total params: 7,193,009
Trainable params: 7,193,009
Non-trainable params: 0
$\endgroup$
1
$\begingroup$

Just think of it as another feature transformation that grasp a set of sequential data group into a convoluted, compressed data. The text is a sequence of words(tokens), and each focus word would have different relationship to another words back and front of it. Conv1D help us to get nice and neat set of features that represent such relationship.

https://realpython.com/python-keras-text-classification/

$\endgroup$
1
$\begingroup$

Conv1d in the first layer would essentially recognize topics (potentially more complicated things as well) but based on the groupings in the word vector space, and over groupings of words based on filter size.

Ex: filter sizes of 2 could be recognizing opinion + negation potentially. "like this", "don't like".

Later layers of these Conv1d will build on the understandings of the earlier layers.

Then you're LSTM can handle the longer term dependencies.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.