Implementing the Dependency Sensitive CNN (DSCNN ) in Keras

Question

I'm currently trying to implement the Dependency Sensitive Convolutional Neural Network for Modeling Documents by Rui Zhang in Keras. For me this is the first network to implement in Keras, so I came up with some questions.

The network looks as follows:

So I'm starting with the input and my thought was that I have to input sequence of sequences, e.g. a list of sentences, where a sentence is a list of words that are word embeddings. The sentences should be padded, so that all sentences have equal length. Also the documents should be padded, so that each document as equal number of sentences.

We will have a batch of the following type (only symbolic -> according to `fit()´):

s_i = np.array(shape=(max_sentence_length, word_embedding_dimensions))
doc_i = [s_1, s_2, s_3, ..., s_(max_sentences_per_doc)]
batch = [doc_1, doc_2, doc_3, ...]

where we_idx_ij is the word embedding index for the j-th word in the i-th sentence of a certain document.

Question 1: Does this make sense?

I still have to think about how I will exactly include the w2v word embeddings, but with the model in Keras I would proceed as follows:

Includes:

Include the needed methods/layers and initialize symbolic constants

import numpy as np
from keras.layers import Activation, Input, Embedding, LSTM, concatenate, AveragePooling1D, MaxPool1D, Conv1D, TimeDistributed

# constants (for testing)
max_sentences_per_document = 10
max_words_per_sentence = 100
w2v_dimensions = 300
vocab_size = 20000
batch_size = 60

Model:

# preparing some shared layers
shared_embedding = Embedding(input_dim=w2v_dimensions, output_dim=vocab_size, weights=[W])  # optional,cause no training
shared_sentence_lstm = TimeDistributed(
    LSTM(input_dim=max_words_per_sentence, return_sequences=True, activation='tanh'),
    input_shape=(max_words_per_sentence, w2v_dimensions)
)
shared_sentence_lstm_2 = LSTM(activation='tanh')

# sentence modeling
sentence_inputs = [Input(shape=(batch_size, max_words_per_sentence, )) for i in range(max_sentences_per_document)]
sentence_modeling = [shared_embedding(sentence_inputs[i]) for i in range(max_sentences_per_document)]
sentence_modeling = [shared_sentence_lstm(sentence_modeling[i]) for i in range(max_sentences_per_document)]
sentence_modeling = [AveragePooling1D()(sentence_modeling[i]) for i in range(max_sentences_per_document)]
sentence_modeling = [shared_sentence_lstm_2(sentence_modeling[i]) for i in range(max_sentences_per_document)]

# document modeling
doc_modeling = concatenate(sentence_modeling)
doc_modeling = Conv1D(filters=100, kernel_size=[3, 4, 5], activation='relu')(doc_modeling)
doc_modeling = MaxPool1D()(doc_modeling)
doc_modeling = Activation('softmax')
doc_modeling.compile(loss='hinge', optimizer='sgd', metrics=['accuracy'])

Question 2:

Does it make sense to have the word embedding already in the batch-data or should I provide the s_i only with an index per word that relates to an index inside the word-embedding matrix? (I guess the latter would make more sense regarding the memory)

Question 3:

If I provide the word embeddings in the batch-data, I wouldn't need the embedding-layer, correct?

Question 4:

Would this network work?

Question 5:

Does this code implement the network as it is proposed in the paper?

Question 6:

Do you have any suggestions to improve the performance?

I hope I'm not too far off the track with this implementation and looking forward to your answers :-) Thanks in advance!

Answer 1

Intermediately I finished the network and it is working great :-) Anyone who is also looking for this network can download it on my Github page.

To refer directly to my questions:

Question 1: The proposed code actually makes partially sense:

• One should use the shared-embedding layer to save memory usage, so this part makes sense
• In the initially proposed code I used TimeDistributed layer, this is the wrong way for the purpose of the network. We actually have to use a simple shared LSTM layer. Time distributed layers are needed for many-to-many networks. For more information about this, I refer to karpathy's blog and Brownlees blog.
• I applied the second LSTM layer before the concatenation, with this I applied two LSTMs to capture the joint meaning of the sentences. That is not the intention of the DSCNN, so we have to apply the one LSTM layer after the concatenation

So far the code looks like:

sentence_inputs = [Input(shape=(max_sentence_len, embedding_dim,), name="input_" + str(i))
                   for i in range(max_sentences_per_doc)]

# LSTMs and Average Pooling (sentence-level)
shared_sentence_lstm = LSTM(units=embedding_dim, return_sequences=True, activation='tanh')
shared_average_pooling = AveragePooling1D(pool_size=max_sentence_len)
sentence_modeling = [shared_sentence_lstm(sentence_inputs[i]) for i in range(max_sentences_per_doc)]
sentence_modeling = [shared_average_pooling(sentence_modeling[i]) for i in range(max_sentences_per_doc)]

doc_modeling = Concatenate(axis=1)(sentence_modeling)
doc_modeling = LSTM(units=embedding_dim, activation='tanh', return_sequences=True)(doc_modeling)

The convolutional layer with multiple filter sizes has to be applied "manually", therefore we run the convolutions in a loop and concatenate the resulting layers after we flattened them (you could also apply GlobalMaxPooling, then you would not have to flatten the layer, but with this you would lose many information of your feature vectors).

conv_blocks = []
for sz in kernel_sizes:
    conv = Convolution1D(filters=filters,
                         kernel_size=sz,
                         padding="valid",
                         activation="relu",
                         strides=1)(doc_modeling)
    conv = MaxPooling1D(pool_size=2)(conv)
    conv = Flatten()(conv)
    conv_blocks.append(conv)
doc_modeling = Concatenate()(conv_blocks) if len(conv_blocks) > 1 else conv_blocks[0]

• After this we can normally apply an activation layer (e.g. softmax or sigmoid, with or without dropout) and compile the model

Question 2 and 3: We need either an embedding directly in the batch-data or the embedding layer. If you implement your network to have the embedding already in the batch data, you have to generate it only once. It works with small datasets and small embedding sizes (small depends on your memory size etc.), but with large datasets (e.g. 2000 words per document and an embedding dimension of 300) you rapidly exceed like 30gb of memory. Therefore it definitely makes sense to use the embedding layer, since it stores each word vector only once. The downside with this embedding layer is, that it has to perform the word index to word vector replacement after each batch, this costs a little bit of runtime, but since it is only a lookup operation this is not very expensive. So be encouraged to use the embedding layer.

Question 4: It would not. Firstly, because of some parameter issues (e.g. Conv1D accepts only one integer for the kernel_size), secondly, it implements another architecture, that makes no sense (cf. question 1, TimeDistributed layer).

Question 5: No :-), see question 4