1
$\begingroup$

While training the IMDB movie review dataset for sentiment analysis, the model will give a memory error if I set the features to more than 20,000. Is there any way to pass the memory error?

$\endgroup$
1
  • $\begingroup$ Get more memory? $\endgroup$ Apr 2, 2018 at 9:02

2 Answers 2

2
$\begingroup$

to answer that question you need to get and give us more information, no chance to advise without knowing where the OOM is thrown and how the memory profile looks like. You may tune the memory usage of your program also in other parts than the part where the error came from, to raise the limit.

Just one hint for a cause that can be resolved without getting more physical memory: Sometimes you can "lazy load" large data sources. Get it record by record or chunk by chunk, then process it, and write results to the sink, before you get the next record into memory. This is done with python generators, watch out for the keyword "yield" that is used instead of return. https://stackoverflow.com/questions/1756096/understanding-generators-in-python

further you may be better off using low level libraries for certain tasks, see i.e. http://www.nltk.org/book/ch06.html 2.4

$\endgroup$
0
$\begingroup$

In order to overcome out of memory error during training, you can either reduce the batch size that would go in for training or if you have a large dataset you can use h5py matrix.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.