I work on preparing the luna16 dataset for feeding into the CNN model, after reading all '.mhd' files and the labels(0, 1) in the annotated CSV file, I get a memory error, I know the problem is because of the data size is need a lot of RAM that I do not have on my computer, is there any solution to that?
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
4
You can use h5py files but due to the fact that they store the dimensions, they take even more disk to be stored and while fetching you have to supply enough memory which is not possible in your case, so you have to read them chunk by chunk. There is a simpler approach. One of pandas's inputs is chunk. try to use it.
answered Jul 30, 2018 at 12:52
-
$\begingroup$ Thank you, you are right, I tried h5py file but during fetching, I get the same problem, Can I feed these chunks directly to CNN model without saving them in h5py file? $\endgroup$– HunarCommented Jul 30, 2018 at 13:21
-
$\begingroup$ Yes, you should specify the chunk, and you have to train your network using batch gradient descent which is common. $\endgroup$ Commented Jul 30, 2018 at 13:22
-
$\begingroup$ Thank you very much, I will read about chunks and try it. $\endgroup$– HunarCommented Jul 30, 2018 at 13:24
-
$\begingroup$ Try to use the last link I've provided :) $\endgroup$ Commented Jul 30, 2018 at 13:25