I'm looking for any tutorials or examples for using either Keras or TensorFlow on large text datasets (at least a few million sentences). I've looked at some of the examples using the
fit_generator() method in Keras, but these are mostly tailored to image classification and involves really minimal preprocessing. I've also looked into using the newer
HDF5Matrix class in Keras, and have continued to run out of memory with batch sizes larger than 20.
Are there any other useful data reading/streaming tools in TensorFlow that could be used to solve this problem? Are there examples for how to do standard NLP preprocessing in such a pipeline in a memory-efficient way before feeding to a model? Any references would be appreciated. Thanks for reading.