1
$\begingroup$

I am currently working on a classification problem that requires me to classify whether an image contains cancerous tissue cells or not. Each image is 50x50x3 pixels, the 3 is for RGB values.

So far I have a pandas dataframe that contains the target value, patient id, image id and the path to the corresponding image.

I can access the image by using 

io.imread(df['path'])

So it is possible for me to loop through all the images to access them. The question now is, where do I store the images so that I can apply principle component analysis on them?

If I were to simply store it in a dataframe it would contain 7500 columns; 1 for each pixel value. My dataset contains 280,000 images. That means my my dataframe would need to be 280,000x7500. I Feel that there is a better way to approach this problem.

Your input to this matter would be highly appreciated.

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ You can always use PIL to read the images and then store them as NumPy arrays. If you are working with TensorFlow/Keras make sure you use ImageDataGenerator class. Using it will avoid memory problems. $\endgroup$
    – Shubham Panchal
    Commented Feb 18, 2020 at 0:17
  • $\begingroup$ Thank you, that is what I ended up using, I flattened the images into the array. Such that each row consists an image containing 7500 columns. If I needed to call a specific image I simply reshape the data, and use the io.imread() function found in sklearn or as you mentioned the PIL read function. $\endgroup$
    – A Merii
    Commented Feb 18, 2020 at 9:48

2 Answers 2

Reset to default
1
$\begingroup$

This might be a bit more complicated.

I normally reuse computer vision and deep learning software to do that. Even if I don't do Deep Learning.

Particularly I use Pytorch, for its bridge with Numpy and pandas. Here is a tutorial.

This allows me to use a GPU if wanted, and to reuse a lot of code since for deep learning and images there is tons of code snippets out there.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ Yes you are 100% correct, this is what I ended up doing. I am using PyTorch dataloaders and so far they have been working splendidly, in fact the tutorial that you linked is one of the best tutorials to do that! $\endgroup$
    – A Merii
    Commented Nov 16, 2020 at 8:59
0
$\begingroup$

Yes pandas won't work well for this. You can look at sparse data formats https://docs.scipy.org/doc/scipy/reference/sparse.html

Or maybe check how it's done in Tensorflow.

Improve this answer
$\endgroup$
1
  • $\begingroup$ I don't think this would be classified as a sparse matrix, maybe it'll be classified as one after PCA but not before. $\endgroup$
    – A Merii
    Commented Feb 17, 2020 at 18:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.