0
$\begingroup$

I have a csv file of size ~11GB contains 2 million rows located in my local disk as my training data. What's the best way to feed the data to a neural network model? I use tf.keras to build the neural network and my RAM doesn't allow me to import as a single np array. Thank you.

$\endgroup$
1
$\begingroup$

You can use generator to read the data if your model uses "Mini-Batch SGD"-like optimization method which just uses a small batch of samples each step. For example,

df_iterator = pandas.read_csv(your_data,chunksize=batch_size,iterator=True)
for small_batch in df_iterator:
#feed it to your model's input
$\endgroup$
0
$\begingroup$

You need mini batch gradient descent. At each training iteration, you feed a batch of data into the model. This is also a great technique to prevent overfitting and making your gradient get stuck into local minima.

Size of the batch is another hyperparameter. Usually batch sizes vary in the 50-250 range, but that's completely up to you. Smaller batches are faster to train, but add more noise to gradient descent; larger batches are more costly to train but they ensure a smoother descent of the gradient.

If you cannot import the whole dataset once, I suggest you to break it into subsets, and start multiple training sessions for each. You will have to complete training in multiple steps, it'll take a bit longer than usual, but it's certainly doable.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.