3
$\begingroup$

I am working on a deep learning problem to detect cancer in images of size 250 x 250. I have hardware limitations and I have been running out of memory.

I decided to convert my images to Matlab formatted files (".mat"), with some improvement; however, I still run out of memory. I have explored some resources that highly recommend using NumPy files (".npy").

It would be costly to convert my images to NumPy files, so I would like to make sure that converting will make a difference. I am not asking for memory enhancement algorithms (e.g. batching), just the memory difference between ".mat" and ".npy" files.

$\endgroup$
1
  • $\begingroup$ Can you post your loading code? Can you post benchmarking, including memory usage? $\endgroup$ Commented Nov 10, 2018 at 18:07

3 Answers 3

1
$\begingroup$

From my research:

  • both can store data in binary format
  • both store the data type of the data
  • I am unsure about compression ratios and load time, which seems to be the subtext of your question.

One thing you don't seem to address is what you are loading the data into, or whether you are considering moving from a MATLAB environment to Python environment or visa-versa.

That said, I found this post useful and thought it may be helpful to you if you have not seen it already. https://stackoverflow.com/a/10997335/3259054 Perhaps you could write a small script to sample some files and see the difference.

Have you considered the HDF5 format? If you are looking to make a change, you might as well test other options too and HDF5 has a lot of momentum towards becoming the de facto standard for scientific computing.

Finally, purely out of a desire to learn, why are you concerned about the file format if you have memory constraints?

$\endgroup$
1
$\begingroup$

One way to reduce in-memory bottlenecks is to more efficiently handle data processing (regardless of the on-disk format).

There are software frameworks designed to improve the training process, especially for loading images. Dask is one such framework to scale existing Python workflows, thus mostly likely it will reduce the memory bottleneck for .npy files relative to .mat files (the only way to be sure is to benchmark).

$\endgroup$
1
$\begingroup$

MATLAB has known for it's memory consumption. So even if you use same data for processing in Python overall system memory utilization will be less in Python.

Based on my experience so far using Python helped me dealing more data with better performance.

One other hand Python have many libraries/Frameworks out of box to further enhance the overall performance and Machine Learning/ Deep Learning (I am not much sure if similar Libraries & Frameworks are available in Matlab also).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.