11
$\begingroup$

I'm trying to use CNN (convolutional neural network) to classify documents. CNN for short text/sentences has been studied in many papers. However, it seems that no papers have used CNN for long text or document.
My problem is that there are too many features from a document. In my dataset, each document has more than 1000 tokens/words. To feed each example to a CNN, I convert each document into a matrix by using word2vec or glove resulting a big matrix. For each matrix, the height is the length of the document, and the width is the size of word embedding vector. My dataset has more than 9000 examples and it takes a lot of time to train the network (a whole week) which makes it difficult to fine-tune parameters.
Another feature extracting method is to use one-hot vector for each word, but this will create very sparse matrices. And of course, this method even takes more time to train than the previous method.
So is there a better method for extracting features without creating large input matrices?
And how should we handle variable length of documents? Currently, I add special strings to make document have the same length, but I don't think it's a good solution.

$\endgroup$
  • 2
    $\begingroup$ Does using TfIdf vectorizer seem suitable? Maybe in combination with word2vec to leave only top x words for each document? $\endgroup$ – Diego Apr 11 '16 at 11:43
  • $\begingroup$ Well, I don't know what Tfldf is. I'm going to check it to see if it works. Thank you $\endgroup$ – lenhhoxung Apr 11 '16 at 12:15
  • $\begingroup$ scikit-learn.org/stable/modules/… here for example $\endgroup$ – Diego Apr 11 '16 at 12:34
  • $\begingroup$ I just check it and I think it doesn't help me. Basically, that helper class creates a matrix for a set of documents. Each row vector (binary or wordcount) in the matrix corresponds to a document, but for CNN, we need a matrix for each document. $\endgroup$ – lenhhoxung Apr 11 '16 at 12:51
  • $\begingroup$ The point was to leave only x non-trivial words per document ranked by their TfIdf. Then use your original encoding to build document matrices. Not sure if this two step approach idea came across. $\endgroup$ – Diego Apr 11 '16 at 17:38
8
$\begingroup$

You could reduce the length of your input data by representing your documents as series of sentence vectors instead of a longer series of word vectors. Doc2vec is one way to do this (each sentence would be a "document").

If you don't want to use Doc2vec, one way to create the sentence vectors would be to average the word vectors for each sentence, giving you a single vector of the same width for each sentence. This may not be as precise as some methods available through Doc2Vec but I have used it with considerable success for topic modeling.

Either way once you have your sentence vectors, line them up in sequence for each document like you're already doing for your word vectors and run then through your model. Because the sequence length for each document is shorter, your model should train more quickly than with word vectors.

By the way, this method could work when scaled up or down to meet your accuracy and speed needs. (e.g. if your CNN still trains too slowly with sentence vectors, you can create paragraph vectors instead).

One way to handle documents of different length is through padding. Your document sequences should all be equal in length to your longest document. So if your longest document is 400 sentences then all document sequences will be 400 vectors in length. Documents shorter than the max length would be padded with vectors filled with zeros.

$\endgroup$
  • $\begingroup$ Interesting idea. I'll try it :) $\endgroup$ – lenhhoxung Sep 21 '16 at 15:59
  • $\begingroup$ May I ask a question? How can I deal with documents that with significantly different lengths(5 sentences/doc, 500 sentences/doc) even I represent them in sentence vectors ? Padding here seems weird... $\endgroup$ – stackunderflow Aug 14 '17 at 4:00
  • 1
    $\begingroup$ In speech processing, some people order the sequences based on its length so that sequence with similar length will be in the same batch. This might work for text sequence. $\endgroup$ – suthee Dec 6 '17 at 23:56
4
$\begingroup$

You could use region embeddings. Rather than converting individual "tokens" to vectors you could use a strategy to convert regions of text to vectors. This approach is used here: https://arxiv.org/abs/1504.01255

If you're not limited to CNN, you could use a hierarchical attention models such as this one: https://www.cs.cmu.edu/~diyiy/docs/naacl16.pdf where you have a pipeline of this kind: word vectors (combined into) sentence vectors (combined into) final document vector

Note that, with this method, you will still have to convert all the word vectors to embeddings, but not all at once.

To handle documents of different lengths, padding/cutting is the only solution so far.

Finally, to increase speed, you could try to reduce the dimension of the text by only including important sections (maybe only the beginning of the document is sufficient to have good classification accuracy)

$\endgroup$
  • 1
    $\begingroup$ Thanks for your reference link. The region embedding is interesting. Concerning variable-sized document, as mentioned in this article arxiv.org/abs/1412.1058 (same author), we can use multiple pooling units instead of padding/cutting. $\endgroup$ – lenhhoxung Jan 4 '18 at 10:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.