I'm looking for any tutorials or examples for using either Keras or TensorFlow on large text datasets (at least a few million sentences). I've looked at some of the examples using the fit_generator() method in Keras, but these are mostly tailored to image classification and involves really minimal preprocessing. I've also looked into using the newer HDF5Matrix class in Keras, and have continued to run out of memory with batch sizes larger than 20.

Are there any other useful data reading/streaming tools in TensorFlow that could be used to solve this problem? Are there examples for how to do standard NLP preprocessing in such a pipeline in a memory-efficient way before feeding to a model? Any references would be appreciated. Thanks for reading.



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.