I am dealing with a multiclass image classification problem with N classes. Particularly interesting is now that a single instance is NOT a single image as you would expect normally, but rather a set of multiple images that belong to each other.

I could theoretically built a model that looks at each image of an instance one after another and classifies them separately (and I actually already did), but I want to create a model that benefits from the interdependence between those images of a single instance.

The reason for that is that there are classes like ATop and ABottom and in my normal image classification set up, it is barely possible to distinguish between those two.

But: In each instance (set of images), there is for example always exactly one image of class ATop and one image of class ABottom. By looking at all images of a class at once, I hope to capture interdependencies between those images that result in a better score on these hard-to-distinguish classes.

My question is now: How can I implement this idea (network structure, data set structure, ...) (in pytorch)?

It might be reasonable to point out that instances can have between 3 to 10 images.


Consider this instance which consists of 4 images:

instance = {image_A, image_B, image_C, image_A'}

I want a model that takes this instance as an input and maps it to

{A, B, C, A'}

Theoretically, I could split this instance into:

instance1 = {image_A}
instance2 = {image_B}
instance3 = {image_C}
instance4 = {image_A'} 

and map them separately to

{A}, {B}, {C}, {A'}

but then, I would lose interdependencies between these images.

A few sidenotes: As pointed out before, a single instance does not always have exactly 4 images. It can have anything between 3 to 10 images and there are also more than these 4 classes (certain classes can also appear more than once in a single instance). Also, the images themselves are not really a sequence. The ordering does not matter. So, the following would be considered identical from human perspective:

instance1 = {image_A, image_B, image_C, image_A'}
instance2 = {image_A, image_A', image_C, image_B}
  • $\begingroup$ What comes to my mind, briefly: take the image classification NN, chop off its final classification layer so that the NN instead returns some vector representation of the image, concat those images for each the images in your instance into one large vector, and feed that vector into a new classification head that you attach to the model. Make the large vector large enough to fit the maximum number of images that appear in an instance, and pad with zeros when there are fewer images than that. Edited to add: you'd only need to train the new head if your classifier is already trained. $\endgroup$
    – Ceph
    Commented Jan 21, 2022 at 18:46
  • $\begingroup$ Yes, this can all be done within a single model architecture - multi head, passing through the same image layers with tied weights, multiple outputs. $\endgroup$
    – Sean Owen
    Commented Apr 27, 2022 at 15:51

2 Answers 2


Finally, I know how to solve this.

The most plain solution would be a RNN-like structure that just acts as a seq2seq model. We can use attention, bidirectional chains etc. to enhance our performance. However, this solution has the clear disadvantage that we insinuate that there is a sequential relationship between the images which is actually not really there.. as pointed out in the question, the ordering of the images in a sample does not matter at all.

As a better solution, we can use a transformer model without positional encoding. As we know, attention is all you need and we can waive the RNN-like structure. Transformer models usually still make use of the positional information in common application areas such as language modeling where the ordering does matter. But in our case, ordering doesn't matter. So, we just take the positional encoding (preprocessing step in the transformer model) away and we have a reasonable architecture where our interdependencies of arbitrary sets of images are captured.


This is sometimes called multi-output.

In PyTorch, it is possible to have multiple inputs and multiple outputs:

import torch.nn as nn

class NeuralNetwork(nn.Module):
  def __init__(self):
    super(NeuralNetwork, self).__init__()
    self.linear1 = nn.Linear()
    self.linear2 = nn.Linear()

  def forward(self, x):
    output1 = self.linear1(x)
    output2 = self.linear2(x)
    return output1, output2
  • $\begingroup$ But as I'd have to set a fixed number of forward calls in my forward method, ...wouldn't that require that all of my instances have the exact same number of images? And also, I honestly don't really understand, how the interdependence between the single images would be captured here. $\endgroup$
    – c0mr4t
    Commented Dec 12, 2021 at 22:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.