I'm currently trying to implement the Dependency Sensitive Convolutional Neural Network for Modeling Documents by Rui Zhang in Keras. For me this is the first network to implement in Keras, so I came up with some questions.

The network looks as follows:

Figure 1: DSCNN (document modeling) by Rui Zhang, copied from his paper.

So I'm starting with the input and my thought was that I have to input sequence of sequences, e.g. a list of sentences, where a sentence is a list of words that are word embeddings. The sentences should be padded, so that all sentences have equal length. Also the documents should be padded, so that each document as equal number of sentences.

We will have a batch of the following type (only symbolic -> according to `fit()´):

s_i = np.array(shape=(max_sentence_length, word_embedding_dimensions))
doc_i = [s_1, s_2, s_3, ..., s_(max_sentences_per_doc)]
batch = [doc_1, doc_2, doc_3, ...]

where we_idx_ij is the word embedding index for the j-th word in the i-th sentence of a certain document.

Question 1: Does this make sense?

I still have to think about how I will exactly include the w2v word embeddings, but with the model in Keras I would proceed as follows:


Include the needed methods/layers and initialize symbolic constants

import numpy as np
from keras.layers import Activation, Input, Embedding, LSTM, concatenate, AveragePooling1D, MaxPool1D, Conv1D, TimeDistributed

# constants (for testing)
max_sentences_per_document = 10
max_words_per_sentence = 100
w2v_dimensions = 300
vocab_size = 20000
batch_size = 60


# preparing some shared layers
shared_embedding = Embedding(input_dim=w2v_dimensions, output_dim=vocab_size, weights=[W])  # optional,cause no training
shared_sentence_lstm = TimeDistributed(
    LSTM(input_dim=max_words_per_sentence, return_sequences=True, activation='tanh'),
    input_shape=(max_words_per_sentence, w2v_dimensions)
shared_sentence_lstm_2 = LSTM(activation='tanh')

# sentence modeling
sentence_inputs = [Input(shape=(batch_size, max_words_per_sentence, )) for i in range(max_sentences_per_document)]
sentence_modeling = [shared_embedding(sentence_inputs[i]) for i in range(max_sentences_per_document)]
sentence_modeling = [shared_sentence_lstm(sentence_modeling[i]) for i in range(max_sentences_per_document)]
sentence_modeling = [AveragePooling1D()(sentence_modeling[i]) for i in range(max_sentences_per_document)]
sentence_modeling = [shared_sentence_lstm_2(sentence_modeling[i]) for i in range(max_sentences_per_document)]

# document modeling
doc_modeling = concatenate(sentence_modeling)
doc_modeling = Conv1D(filters=100, kernel_size=[3, 4, 5], activation='relu')(doc_modeling)
doc_modeling = MaxPool1D()(doc_modeling)
doc_modeling = Activation('softmax')
doc_modeling.compile(loss='hinge', optimizer='sgd', metrics=['accuracy'])

Question 2:

Does it make sense to have the word embedding already in the batch-data or should I provide the s_i only with an index per word that relates to an index inside the word-embedding matrix? (I guess the latter would make more sense regarding the memory)

Question 3:

If I provide the word embeddings in the batch-data, I wouldn't need the embedding-layer, correct?

Question 4:

Would this network work?

Question 5:

Does this code implement the network as it is proposed in the paper?

Question 6:

Do you have any suggestions to improve the performance?

I hope I'm not too far off the track with this implementation and looking forward to your answers :-) Thanks in advance!


1 Answer 1


Intermediately I finished the network and it is working great :-) Anyone who is also looking for this network can download it on my Github page.

To refer directly to my questions:

Question 1: The proposed code actually makes partially sense:

  • One should use the shared-embedding layer to save memory usage, so this part makes sense
  • In the initially proposed code I used TimeDistributed layer, this is the wrong way for the purpose of the network. We actually have to use a simple shared LSTM layer. Time distributed layers are needed for many-to-many networks. For more information about this, I refer to karpathy's blog and Brownlees blog.
  • I applied the second LSTM layer before the concatenation, with this I applied two LSTMs to capture the joint meaning of the sentences. That is not the intention of the DSCNN, so we have to apply the one LSTM layer after the concatenation

So far the code looks like:

sentence_inputs = [Input(shape=(max_sentence_len, embedding_dim,), name="input_" + str(i))
                   for i in range(max_sentences_per_doc)]

# LSTMs and Average Pooling (sentence-level)
shared_sentence_lstm = LSTM(units=embedding_dim, return_sequences=True, activation='tanh')
shared_average_pooling = AveragePooling1D(pool_size=max_sentence_len)
sentence_modeling = [shared_sentence_lstm(sentence_inputs[i]) for i in range(max_sentences_per_doc)]
sentence_modeling = [shared_average_pooling(sentence_modeling[i]) for i in range(max_sentences_per_doc)]

doc_modeling = Concatenate(axis=1)(sentence_modeling)
doc_modeling = LSTM(units=embedding_dim, activation='tanh', return_sequences=True)(doc_modeling)

The convolutional layer with multiple filter sizes has to be applied "manually", therefore we run the convolutions in a loop and concatenate the resulting layers after we flattened them (you could also apply GlobalMaxPooling, then you would not have to flatten the layer, but with this you would lose many information of your feature vectors).

conv_blocks = []
for sz in kernel_sizes:
    conv = Convolution1D(filters=filters,
    conv = MaxPooling1D(pool_size=2)(conv)
    conv = Flatten()(conv)
doc_modeling = Concatenate()(conv_blocks) if len(conv_blocks) > 1 else conv_blocks[0]
  • After this we can normally apply an activation layer (e.g. softmax or sigmoid, with or without dropout) and compile the model

Question 2 and 3: We need either an embedding directly in the batch-data or the embedding layer. If you implement your network to have the embedding already in the batch data, you have to generate it only once. It works with small datasets and small embedding sizes (small depends on your memory size etc.), but with large datasets (e.g. 2000 words per document and an embedding dimension of 300) you rapidly exceed like 30gb of memory. Therefore it definitely makes sense to use the embedding layer, since it stores each word vector only once. The downside with this embedding layer is, that it has to perform the word index to word vector replacement after each batch, this costs a little bit of runtime, but since it is only a lookup operation this is not very expensive. So be encouraged to use the embedding layer.

Question 4: It would not. Firstly, because of some parameter issues (e.g. Conv1D accepts only one integer for the kernel_size), secondly, it implements another architecture, that makes no sense (cf. question 1, TimeDistributed layer).

Question 5: No :-), see question 4

  • 1
    $\begingroup$ It's a bit of an over-long question and short answer. However, thank you for not forgetting the question once you already had your results, and providing this self answer. If have time to come back and expand this answer a little bit more (maybe two or three full sentences per question point) it would be even better. $\endgroup$ Jul 7, 2017 at 17:56
  • $\begingroup$ Thank you for answering your own question. I have some suggestions for improving this answer. We discourage link-only answers. It looks like your answers to Questions 1 and 4 are link-only answers (this answer doesn't make sense without looking at external material), and you haven't really answered Questions 2 and 5, leaving only the answer to Question 3, which consists of a single word "Exact" -- and I'm not sure what that means. $\endgroup$
    – D.W.
    Jul 8, 2017 at 5:04
  • $\begingroup$ This would be a better answer if you edited it to summarize your approach to Question 1 and 4 in the answer, explained what you mean by "Exact", and provided some justification for your statements about Questions 2 and 3. $\endgroup$
    – D.W.
    Jul 8, 2017 at 5:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.