I have a csv file of size ~11GB contains 2 million rows located in my local disk as my training data. What's the best way to feed the data to a neural network model? I use tf.keras to build the neural network and my RAM doesn't allow me to import as a single np array. Thank you.
You can use generator to read the data if your model uses "Mini-Batch SGD"-like optimization method which just uses a small batch of samples each step. For example,
df_iterator = pandas.read_csv(your_data,chunksize=batch_size,iterator=True) for small_batch in df_iterator: #feed it to your model's input
You need mini batch gradient descent. At each training iteration, you feed a batch of data into the model. This is also a great technique to prevent overfitting and making your gradient get stuck into local minima.
Size of the batch is another hyperparameter. Usually batch sizes vary in the 50-250 range, but that's completely up to you. Smaller batches are faster to train, but add more noise to gradient descent; larger batches are more costly to train but they ensure a smoother descent of the gradient.
If you cannot import the whole dataset once, I suggest you to break it into subsets, and start multiple training sessions for each. You will have to complete training in multiple steps, it'll take a bit longer than usual, but it's certainly doable.