# Classification problem with many images per instance

I am working in the following kind of classification problem: I have to classify every instance as class A or class B using many images of the instance. That is, every training example has not one image (which is the usual thing in image classification), but many images, and the number of images for every training instance is not fixed. That is, instance 1 can have 3 images, and according to these images we have to classify it as A or B, and instance 2 can have instead 5 images.

As any machine learning problem, I am provided with many labelled images and I have to build a classifier.

Although ideas are also welcome, I am looking for a documented way to attack this kind of problem (Kaggles, papers or books, mainly).

My main idea was the following: train a model $f$ that given one image gives a probability of that image being of class A. Then, for every training instance, evaluate $f$ in every image of the instance and compute statistics (aggregate) of the distribution of these probabilities, as the mean, median, maximum and minimum. Then, train a model $g$ that has as inputs these aggregates and use the composition of $f$, aggregates and $g$ as the final model. This idea is a bit simple so I am looking for something better.

• Does A look like B from some angles or object poses, under some lighting, or in any variations of the image? Another way to put this: Would a human expert be more likely to mis-classify any single image in your image sets due to how the image was presented, and are such presentations likely to occur in your production data? In addition, what is the class distribution - is it skewed towards A or B? – Neil Slater Sep 11 '18 at 18:03
• Actually, A are pictures of hotels and B are pictures of hostels. So they are not object poses, they are pictures of different parts of a common place. So, a human could might miss-classify one image in every instance but should be able to say if it is a hotel or hostel by looking at all the pictures of the instance. – David Masip Sep 11 '18 at 18:06
• Right, so in some cases the individual images might be ambiguous to a human, but you claim that a full set of images is rarely ambiguous? – Neil Slater Sep 11 '18 at 18:08
• Exactly, that is what I meant – David Masip Sep 11 '18 at 18:09

This might be a hack, but, have you considered repeating some images to make every instance contain a fix number of images? Let's say you have images A, B, C, D, E, F, G, H, and I. Your dataset might look like this:

• A , B
• C
• D, E, F
• G
• H, I

Make a function that will randomly select one of the image in an instance and then use that as placeholder. From our example, we now modify our dataset into:

• A, B, A
• C, C, C
• D, E, F
• G, G, G
• H, I, I

Then, from here, you can use a convolutional neural network with 3 inputs. Here is an example of a CNN with multiple inputs: Multi-Input Convolutional Neural Network for Flower Grading - images of same flowers taken at different angles.

Also, you can shuffle the images in each instance to produce more training samples.

• Have you seen any implementation of a multi-input CNN in keras/tensorflow/pytorch? – David Masip Sep 20 '18 at 8:11
• – atmarges Sep 22 '18 at 9:56
• Have you tried keras' functional API? Here's a good example: keras.io/getting-started/functional-api-guide/… . And here is a notebook that uses multi-input CNN: kaggle.com/hireme/two-inputs-neural-network-using-keras . Just focus on input #7. Basically, you just create 2 (or more) input tensors. Then modify the model as you want. You can link the 2 input to separate CNN then concatenate the results of the two. Or you can also link them to two CNN with shared weights. Or maybe add attention mechanism to give different weights to the output – atmarges Sep 22 '18 at 10:10
• Thanks, really appreciate the creative idea and the implementation link – David Masip Sep 22 '18 at 10:50
• You're welcome. I'm glad I could help. :) – atmarges Sep 22 '18 at 11:04

I assume every instance is a grouped data of either hotels or hostels and both. I think this paper discusses your problem, provides solutions and comparisons between different frameworks.

https://pdfs.semanticscholar.org/7f13/9a109d8720659f676d6f2b99ecf289433b29.pdf

Instead of characterising individual pictures, we could characterise the groups using the distribution of all the observations in an instance. We can assume that the pictures of both hostels and hotels are drawn from a parameterised distribution. The main idea suggests to train every image to calculate the probability of belonging in a class. When you calculate the individual probabilities, the aggregation of these in a group would lead to a higher misclassification rate. This method leverages extra information in the instance. For example, if these instances are different neighbourhoods of an area and all the pictures are from different neighbourhoods, then it becomes harder to predict the class of the instance but with DBA, a network can also learn about these neighbourhoods in particular.

• It would be helpful if you could summarise the points or at least the main conclusions contained in the linked paper. – n1k31t4 Sep 20 '18 at 9:12

So on occurrence of such problem, one approach that could be followed is stacking up all the images from one sub-class one after the other, to form a high dimensional picture and pad the rest of the space with blank tensors.

Lets say, we have A,B,C,D,E,F,G,H,I,J images scattered as follows:

hotel1: A,B,C
hotel2: D,E,F,G,H
hotel3: I,J

Here I would simply stack each image in every sub class like channels into a vector of shape (number_of_subclass, maximum_images_subclass) to obtain a higher dimension image over just (height, width, channels).

hotel1: A,B,C,0,0
hotel2: D,E,F,G,H
hotel3: I,J,0,0,0

So the resulting vector will be of shape (number_of_subclass, maximum_images_subclass, height, width, channels).

Now the obtained tensors could easily be fed into the model as per requirement.

Use logistic regression. It accounts for the uncertainty in image labeling; new images will be classified as A or B with a probability associated with it.