I am currently working on a classification problem that requires me to classify whether an image contains cancerous tissue cells or not. Each image is 50x50x3 pixels, the 3 is for RGB values.

So far I have a pandas dataframe that contains the target value, patient id, image id and the path to the corresponding image.

I can access the image by using


So it is possible for me to loop through all the images to access them. The question now is, where do I store the images so that I can apply principle component analysis on them?

If I were to simply store it in a dataframe it would contain 7500 columns; 1 for each pixel value. My dataset contains 280,000 images. That means my my dataframe would need to be 280,000x7500. I Feel that there is a better way to approach this problem.

Your input to this matter would be highly appreciated.

  • 1
    $\begingroup$ You can always use PIL to read the images and then store them as NumPy arrays. If you are working with TensorFlow/Keras make sure you use ImageDataGenerator class. Using it will avoid memory problems. $\endgroup$ – Shubham Panchal Feb 18 at 0:17
  • $\begingroup$ Thank you, that is what I ended up using, I flattened the images into the array. Such that each row consists an image containing 7500 columns. If I needed to call a specific image I simply reshape the data, and use the io.imread() function found in sklearn or as you mentioned the PIL read function. $\endgroup$ – A Merii Feb 18 at 9:48

Yes pandas won't work well for this. You can look at sparse data formats https://docs.scipy.org/doc/scipy/reference/sparse.html

Or maybe check how it's done in Tensorflow.

| improve this answer | |
  • $\begingroup$ I don't think this would be classified as a sparse matrix, maybe it'll be classified as one after PCA but not before. $\endgroup$ – A Merii Feb 17 at 18:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.