# What is the best way to store images in python for machine learning

I am currently working on a classification problem that requires me to classify whether an image contains cancerous tissue cells or not. Each image is 50x50x3 pixels, the 3 is for RGB values.

So far I have a pandas dataframe that contains the target value, patient id, image id and the path to the corresponding image.

I can access the image by using

io.imread(df['path'])


So it is possible for me to loop through all the images to access them. The question now is, where do I store the images so that I can apply principle component analysis on them?

If I were to simply store it in a dataframe it would contain 7500 columns; 1 for each pixel value. My dataset contains 280,000 images. That means my my dataframe would need to be 280,000x7500. I Feel that there is a better way to approach this problem.

Your input to this matter would be highly appreciated.

• You can always use PIL to read the images and then store them as NumPy arrays. If you are working with TensorFlow/Keras make sure you use ImageDataGenerator class. Using it will avoid memory problems. – Shubham Panchal Feb 18 at 0:17
• Thank you, that is what I ended up using, I flattened the images into the array. Such that each row consists an image containing 7500 columns. If I needed to call a specific image I simply reshape the data, and use the io.imread() function found in sklearn or as you mentioned the PIL read function. – A Merii Feb 18 at 9:48

This might be a bit more complicated.

I normally reuse computer vision and deep learning software to do that. Even if I don't do Deep Learning.

Particularly I use Pytorch, for its bridge with Numpy and pandas. Here is a tutorial.

This allows me to use a GPU if wanted, and to reuse a lot of code since for deep learning and images there is tons of code snippets out there.

• Yes you are 100% correct, this is what I ended up doing. I am using PyTorch dataloaders and so far they have been working splendidly, in fact the tutorial that you linked is one of the best tutorials to do that! – A Merii Nov 16 at 8:59

Yes pandas won't work well for this. You can look at sparse data formats https://docs.scipy.org/doc/scipy/reference/sparse.html

Or maybe check how it's done in Tensorflow.

• I don't think this would be classified as a sparse matrix, maybe it'll be classified as one after PCA but not before. – A Merii Feb 17 at 18:41