Document classification using convolutional neural network

Question

I'm trying to use CNN (convolutional neural network) to classify documents. CNN for short text/sentences has been studied in many papers. However, it seems that no papers have used CNN for long text or document.

My problem is that there are too many features from a document. In my dataset, each document has more than 1000 tokens/words. To feed each example to a CNN, I convert each document into a matrix by using word2vec or glove resulting a big matrix. For each matrix, the height is the length of the document, and the width is the size of word embedding vector. My dataset has more than 9000 examples and it takes a lot of time to train the network (a whole week) which makes it difficult to fine-tune parameters.

Another feature extracting method is to use one-hot vector for each word, but this will create very sparse matrices. And of course, this method even takes more time to train than the previous method.

So is there a better method for extracting features without creating large input matrices?
And how should we handle variable length of documents? Currently, I add special strings to make document have the same length, but I don't think it's a good solution.

Answer 1

You could reduce the length of your input data by representing your documents as series of sentence vectors instead of a longer series of word vectors. Doc2vec is one way to do this (each sentence would be a "document").

If you don't want to use Doc2vec, one way to create the sentence vectors would be to average the word vectors for each sentence, giving you a single vector of the same width for each sentence. This may not be as precise as some methods available through Doc2Vec but I have used it with considerable success for topic modeling.

Either way once you have your sentence vectors, line them up in sequence for each document like you're already doing for your word vectors and run then through your model. Because the sequence length for each document is shorter, your model should train more quickly than with word vectors.

By the way, this method could work when scaled up or down to meet your accuracy and speed needs. (e.g. if your CNN still trains too slowly with sentence vectors, you can create paragraph vectors instead).

One way to handle documents of different length is through padding. Your document sequences should all be equal in length to your longest document. So if your longest document is 400 sentences then all document sequences will be 400 vectors in length. Documents shorter than the max length would be padded with vectors filled with zeros.

Answer 2

You could use region embeddings. Rather than converting individual "tokens" to vectors you could use a strategy to convert regions of text to vectors. This approach is used here.

If you're not limited to CNN, you could use a hierarchical attention models such as this one where you have a pipeline of this kind:

word vectors (combined into) sentence vectors (combined into) final document vector

Note that, with this method, you will still have to convert all the word vectors to embeddings, but not all at once.

To handle documents of different lengths, padding/cutting is the only solution so far.

Finally, to increase speed, you could try to reduce the dimension of the text by only including important sections (maybe only the beginning of the document is sufficient to have good classification accuracy)