What should the output sizing be for a class that returns multiple image arrays for a dataloader

Question

I have a custom image class (mainly borrowed from examples) that takes in an image of size 3x244x244 for use in a VGG model and returns augmented versions (rotations of 90,180,270 and an Hflip). Previously, I used this class to get one image and return that and the dataloader and everything worked as expected. When I try to output multiple image arrays, I'm not sure what the proper setup should be for use in the batching (I end up with a list of four image arrays) as one batch(?), which needs to be broken out into individual arrays (but it's still in the dataloader loop).

    def __init__(self, patch_data,file_count=1,transform=None, target_transform=None):

    def __getitem__(self, index):
        image = self.patch_data
        image.astype(float)

        #Normalize the data to 0,1 from 2^16
        image = self.image_normalize(image) #image/65535.0 #image_normalize(image)
        output90, output180, output270, outputflip = create_augmented_data(image,flip = 1,rot90=1,rot180=1,rot270=1)
        out_data = [output90,output180, output270, outputflip]
        sample = {"image": out_data} #, "label": label}
            #sample = file_name
        return sample

to run the data into the model, I have:

        for i, data in enumerate(dataloader_all, 0):
            inputs = data['image']#.type(FloatTensor)
            outputs = model_vgg16(inputs)

The conv2d sizing complains because it's setup to expect a tensor of 3x244x244, but it is getting a list of multiples of those. Since I'm in that dataloader for loop, I'm unclear on what might be needed to get all 4 arrays in the list to be properly batched into the model. Since this is using the eval mode of the model, I "think" I could gear it to run for each item in the list separately, but it seems that I had something wrong way before altering it that way. Any ideas?

Answer 1

You are almost there. You just need to perform the batch inference correctly. So, while model inference you need to convert the list of images to a single tensor, as follows -

for i, data in enumerate(dataloader_all, 0):
    inputs = torch.stack(data['image'])
    outputs = model_vgg16(inputs)

Your code might be creating a tensor of list of images, while the model expects tensor of tensors (tensor of image tensors). You need to give a tensor of shape (num_images,3,244,244) as an input to your model. Where num_images is batch-size (4 or more in your case). You should go ahead and try to print and see the tensor shapes.

Also, just make sure your dataloader is converting your images to tensor. I think you are already doing this because you mentioned you already made single image inference.