2
$\begingroup$

I work on preparing the luna16 dataset for feeding into the CNN model, after reading all '.mhd' files and the labels(0, 1) in the annotated CSV file, I get a memory error, I know the problem is because of the data size is need a lot of RAM that I do not have on my computer, is there any solution to that?

$\endgroup$
2
$\begingroup$

You can use h5py files but due to the fact that they store the dimensions, they take even more disk to be stored and while fetching you have to supply enough memory which is not possible in your case, so you have to read them chunk by chunk. There is a simpler approach. One of pandas's inputs is chunk. try to use it.

$\endgroup$
4
  • $\begingroup$ Thank you, you are right, I tried h5py file but during fetching, I get the same problem, Can I feed these chunks directly to CNN model without saving them in h5py file? $\endgroup$
    – Hunar
    Jul 30 '18 at 13:21
  • $\begingroup$ Yes, you should specify the chunk, and you have to train your network using batch gradient descent which is common. $\endgroup$
    – Media
    Jul 30 '18 at 13:22
  • $\begingroup$ Thank you very much, I will read about chunks and try it. $\endgroup$
    – Hunar
    Jul 30 '18 at 13:24
  • $\begingroup$ Try to use the last link I've provided :) $\endgroup$
    – Media
    Jul 30 '18 at 13:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.