Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
I'm getting an error while processing 0.2 million of text data. I'm using CNN text classification in tensorflow. Output raw data shape is 204177x22000. while passing to numpy.array(out_raw), here it is consuming 100% memory(Using 8GB RAM). Tried with data in batch but didn't work. If i need to increase my RAM size then kindly help me out with formula.
What are the methods to take care of this problem statement?
Like @Sagar said, you could convert your pipeline to pyspark (So Spark with python API), and you can set your memory usage to not go above 1G of RAM for example and this will be faster because of the parallelization.
You can try to use generator with Tensorflow, that will fit back and forth your data so it never explode your RAM.
If the issue is before, like getting the data into a variable, you could use Dask. Which is simply a distributed Pandas that helps you fit your data in RAM by processing them by batch
np.ndarray(shape=(211500,23500), dtype=float, order='F'), in this case also the memory error occurs. how to debug this problem?
$\endgroup$
Dec 24, 2018 at 13:39
Are you running in jupyter notebook. There are several options:
