2
$\begingroup$

I'm getting an error while processing 0.2 million of text data. I'm using CNN text classification in tensorflow. Output raw data shape is 204177x22000. while passing to numpy.array(out_raw), here it is consuming 100% memory(Using 8GB RAM). Tried with data in batch but didn't work. If i need to increase my RAM size then kindly help me out with formula.

What are the methods to take care of this problem statement?

$\endgroup$
  • $\begingroup$ how about pyspark. Hope this will help else you need to increase ram $\endgroup$ – Sagar Dec 21 '18 at 7:30
2
$\begingroup$

Like @Sagar said, you could convert your pipeline to pyspark (So Spark with python API), and you can set your memory usage to not go above 1G of RAM for example and this will be faster because of the parallelization.

You can try to use generator with Tensorflow, that will fit back and forth your data so it never explode your RAM.

If the issue is before, like getting the data into a variable, you could use Dask. Which is simply a distributed Pandas that helps you fit your data in RAM by processing them by batch

$\endgroup$
  • $\begingroup$ I'll try these methods. $\endgroup$ – Ritesh Dubal Dec 24 '18 at 13:36
  • $\begingroup$ If i only use numpy and shape is np.ndarray(shape=(211500,23500), dtype=float, order='F'), in this case also the memory error occurs. how to debug this problem? $\endgroup$ – Ritesh Dubal Dec 24 '18 at 13:39
1
$\begingroup$

Are you running in jupyter notebook. There are several options:

  • Convert your data to feather or hd5 format
  • You can opt amazon machine learning EC2 instances
  • create an account in Paperspace.com for morecomputing power
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.