How can I preprocess multi-page image inputs in a theano/lasagne network?

Question

I am trying to classify multi-page documents using a convolutional neural network (CNN). The content of each page in the corpus contains only text (i.e., no photographs or icons), and different documents may have a different shape (height and width). I'd like my classification approach to use all pages in each document, rather than just the first page.

As far as I know, the input to a CNN (like theano) needs a standardized shape. My first thought was to create a single image array that has all the pages concatenated. But I would then have to resize/zero pad all concatenated pages to match the length of the document with the largest page count, and use that as my lowest height and width for the set of concatenated pages. If I don't use this strategy I risk losing resolution on the words in the image, but that is a giant input vector.

I feel like splitting each document's pages into separate samples would be a better approach to standardize input preprocessing, but I'm at a loss for how to classify the whole document if I'm just training on loose pages of each document. Can anyone advise on me on this?

Answer 1

First point is that convolutional neural networks would be incredibly expensive to train on images that large. The minimum size you could allow would be num_pixels_per_letter x num_letters_per_row_or_col.

It sounds like you want to achieve two tasks: OCR then document classification. Given that you have predominantly text in these documents, you would almost certainly be better served by using a more a traditional method for OCR. If you want or need to use NNs for the OCR piece, you will almost certainly need to use a sliding window. For inspiration on how you might do that, you could look to Recurrent Neural Networks that process images in sequences

Once you have the text there are again much simpler methods to deal with text than NNs, but they can achieve great results. To deal with different document lengths, you can use an RNN architecture like LSTM. Or you can use paragraph vectors, which do most of the heavy lifting for you and give you an N-dimensional representation of your text.