0
$\begingroup$

TLDR: What is a best way to add borders to rectangular images with widely variable aspect ratios (from 0.5 to 2.0) to later use them as training data in StyleGAN2 with data resolution of 1024x1024 and preserve the benefit of mirror augment

So I have a custom dataset consisting of ~70k images of variable resolutions and aspect ratios of the images ranging from 0.5 to 2.0. I want to use this dataset to train a StyleGAN2 model but it requires a square images (and even if it didn't, my images are still of different aspect ratios). One of the most common solutions is to just crop the images down to squares but I cant use this approach since >90% of the image area contains an important information and cropping it out will render the whole process useless.

As an example consider a following image of a street map (not my actual data, but illustrates the problem):

street map

It has an aspect W/H ratio of about 0.5, and suppose all of the pixels contain a useful information so we don't want to crop the image.

Now the possible options to turn this image of a street map to a square are:

  1. Adding a constant border:

street map with black border

Now the image is a square but contains a hard boundary between the content and the border which is probably not ideal at all.

  1. Adding a replicated border (I am using OpenCV's BORDER_REPLICATE):

street map with replicated border

Probably a better solution than a constant border but the lines from pixel replication will probably make the NN training more complicated since now a large portion of images contains an alien lines which are basically noise compared to the image data itself

  1. Adding a reflected border (BORDER_REFLECT)

street map with reflected border

Now the border contains basically identical information to the image, but now it seems like doing a mirror augmentation in StyleGAN2 will produce less value since now images contain a reflection on their own.

I am also considering shifting the image to the side so that the border is only added to the right, but it will probably cause the same issue with mirror augmenting, but I am not sure

  1. Warping an image at the border (BORDER_WRAP):

street map with wrapped border

May also be a good option, but again it contains a hard boundary (like in option 1 with constant border) between the image and the data. Another thing (which applies to the reflected borders too) is that the extra images placed as borders will in most cases be cropped (like if image occupies 70% of a square and we fill remaining 30% with reflections, they will not be full and will only partially reflect/wrap the original image).

  1. Resizing the image:

resized street map

One of the last options I was thinking of. The issue is that I will have to then manually find the aspect ratios for the output images from the StyleGAN2 which I am sure will be a pain, not to mention how such data manipulation will affect the training process (don't think it will help to be honest)

As a result

I am asking for any help in this question of what is a best way to turn rectangular image to a square ones. I understand the there will brobably won't be the best approach but I am looking for some insight on what is a better way to do this. I would be happy to try them all on my own, but training the StyleGAN2 is a lengthy process so I don't really want to waste a couple of attempts for bad results, so I would be really thankful if someone can provide some information on this.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.