This question is a further step of this question.
My data inputs are tens of .csv files, I have already read csv input data until the following format:
# train_x is data, train_y is label print(train_x.shape) # (2000000,10,100) 3D array print(train_y.shape) # (2000000,) labels
I already can fit & evaluate them using:
model.fit(train_x, train_y, batch_size=32, epochs=10) model.evaluate(train_x, train_y)
It works well if the dataset is LESS than RAM size. But if dataset is too BIG then "large dataset do not fit the memory". Most online suggestions are to use fit_generator( ) instead of fit( ) (also suggested from keras website).
fit_generator(generator, steps_per_epoch=None, epochs=1, verbose=1, callbacks=None, validation_data=None, validation_steps=None, validation_freq=1, class_weight=None, max_queue_size=10, workers=1, use_multiprocessing=False, shuffle=True, initial_epoch=0)
How to write a generator function (the 1st parameter of fit_generator)?
- I only know the generator function aims to feed data batch by batch.
As the name suggests, the .fit_generator function assumes there is an underlying function that is generating the data for it.
- What should be included in this geneator function? What should be returned? Any related example?