CNN Architecture for Multiple Instance Learning

Question

I have a binary classification problem where I have a bag of documents (image files) that I need to classify - the bag that is, not the individual document. However, a bag can have a different number of documents and the combination of documents will determine the classification. E.g.

Bag1: A, B, C     label = Pass
Bag2: D, E                Fail
Bag3: F                   Pass

In Bag1, the presence of A, B, or C alone may not be enough to Pass, but together they do. In Bag3, F itself might be enough to pass.

I have already done OCR to represent all the text at the Bag level. For the images, all I can think is to average the pixel values of each image in the Bag to represent it as a single image, and then train with a CNN.

Is there any better architecture or method for handling something like this?

Answer 1

As I understand there is some information that needs to be present in order for the example to "pass". I have a few questions as well as a suggestion. First off do the information come in different forms? Meaning, different fonts, different kind of documents for the same information etc? Is the cnn mandatory? Any more info, regarding the domain of the application or the nature of the problem that you could share?

Suggestions:

• Since you have done the OCR I would suggest a pipeline of different models. Process the outputs of the OCR so they can create a secondary dataset in order to employ NLP. Meaning decode the OCRs to produce plaintext outputs, which would serve as the secondary data-set.

• Another thing you could try is to identify the category of each document just like any other multi-class classification. After that you could, employ another classifier based on which categories pass your test (if there 's a discrete number) and continue like so.
  Example: Identify Type A through Type F documents. Set the combinations that would serve as true labels, e.g occurrence of Type F, occurrence of Type A,B,C (any other combination would be False) and then train.

Finally there's a paper on a different domain, though an approach similar to theirs could be followed. They present a Bag-of-Visual Words (BOV) method for object-based classification in land-use/cover mapping.

...we can characterize an image by a histogram of visual-word count [6]. The visual vocabulary provides a “midlevel” representation which helps to bridge the huge semantic gap between the low-level features extracted from an image and the high-level concepts to be categorized

Object Classification of Aerial Images With Bag-of-Visual Words