I'm currently trying to implement the Dependency Sensitive Convolutional Neural Network for Modeling Documents by Rui Zhang in Keras. For me this is the first network to implement in Keras, so I came up with some questions.
The network looks as follows:
So I'm starting with the input and my thought was that I have to input sequence of sequences, e.g. a list of sentences, where a sentence is a list of words that are word embeddings. The sentences should be padded, so that all sentences have equal length. Also the documents should be padded, so that each document as equal number of sentences.
We will have a batch of the following type (only symbolic -> according to `fit()´):
s_i = np.array(shape=(max_sentence_length, word_embedding_dimensions)) doc_i = [s_1, s_2, s_3, ..., s_(max_sentences_per_doc)] batch = [doc_1, doc_2, doc_3, ...]
we_idx_ij is the word embedding index for the j-th word in the i-th sentence of a certain document.
Question 1: Does this make sense?
I still have to think about how I will exactly include the w2v word embeddings, but with the model in Keras I would proceed as follows:
Include the needed methods/layers and initialize symbolic constants
import numpy as np from keras.layers import Activation, Input, Embedding, LSTM, concatenate, AveragePooling1D, MaxPool1D, Conv1D, TimeDistributed # constants (for testing) max_sentences_per_document = 10 max_words_per_sentence = 100 w2v_dimensions = 300 vocab_size = 20000 batch_size = 60
# preparing some shared layers shared_embedding = Embedding(input_dim=w2v_dimensions, output_dim=vocab_size, weights=[W]) # optional,cause no training shared_sentence_lstm = TimeDistributed( LSTM(input_dim=max_words_per_sentence, return_sequences=True, activation='tanh'), input_shape=(max_words_per_sentence, w2v_dimensions) ) shared_sentence_lstm_2 = LSTM(activation='tanh') # sentence modeling sentence_inputs = [Input(shape=(batch_size, max_words_per_sentence, )) for i in range(max_sentences_per_document)] sentence_modeling = [shared_embedding(sentence_inputs[i]) for i in range(max_sentences_per_document)] sentence_modeling = [shared_sentence_lstm(sentence_modeling[i]) for i in range(max_sentences_per_document)] sentence_modeling = [AveragePooling1D()(sentence_modeling[i]) for i in range(max_sentences_per_document)] sentence_modeling = [shared_sentence_lstm_2(sentence_modeling[i]) for i in range(max_sentences_per_document)] # document modeling doc_modeling = concatenate(sentence_modeling) doc_modeling = Conv1D(filters=100, kernel_size=[3, 4, 5], activation='relu')(doc_modeling) doc_modeling = MaxPool1D()(doc_modeling) doc_modeling = Activation('softmax') doc_modeling.compile(loss='hinge', optimizer='sgd', metrics=['accuracy'])
Does it make sense to have the word embedding already in the batch-data or should I provide the
s_i only with an index per word that relates to an index inside the word-embedding matrix? (I guess the latter would make more sense regarding the memory)
If I provide the word embeddings in the batch-data, I wouldn't need the embedding-layer, correct?
Would this network work?
Does this code implement the network as it is proposed in the paper?
Do you have any suggestions to improve the performance?
I hope I'm not too far off the track with this implementation and looking forward to your answers :-) Thanks in advance!