how to build an images dataset efficiently

Question

I have several image datasets and I am going to combine all of them into 1 giant dataset I decided to load the images which are in different formats (SVG, PIL images, etc ) and save them as png but this took forever, there any alternative solution?

Answer 1

Yes,

You can use Image processing tools, Such as ImageMagick[1], GraphicsMagick[2](Similar to ImageMagick), and PIL[3] (Python Imaging Library or Known as Pillow). These tools can convert images fastly.

Here is an example of a Python Imaging Library code[4]:

from PIL import Image

im1 = Image.open(r'path where the JPG is stored\file name.jpg')
im1.save(r'path where the PNG will be stored\new file name.png

Reference:

1. https://imagemagick.org/
2. http://www.graphicsmagick.org/
3. https://pillow.readthedocs.io/
4. https://datatofish.com/jpeg-to-png-python/

Answer 2

There are a couple of different options to solve this problem:

1. Create a custom Dataloader class - Depending on the modeling framework, different file formats can be read and modified at the loading stage. This is a useful option because most modeling packages assume all images are the same size/channels.

2. Resave all images - Open, convert, and resave all images using a library like Python Imaging Library (PIL). This option can be speed-up with multiprocessing.