How to write a generator for keras fit_generator?

Question

This question is a further step of this question.

My data inputs are tens of .csv files, I have already read csv input data until the following format:

# train_x is data, train_y is label
print(train_x.shape)       # (2000000,10,100)  3D array
print(train_y.shape)       # (2000000,)    labels

I already can fit & evaluate them using:

model.fit(train_x, train_y, batch_size=32, epochs=10)
model.evaluate(train_x, train_y)

It works well if the dataset is LESS than RAM size. But if dataset is too BIG then "large dataset do not fit the memory". Most online suggestions are to use fit_generator( ) instead of fit( ) (also suggested from keras website).

fit_generator(generator, steps_per_epoch=None, epochs=1, verbose=1, callbacks=None, validation_data=None, validation_steps=None, validation_freq=1, class_weight=None, max_queue_size=10, workers=1, use_multiprocessing=False, shuffle=True, initial_epoch=0)

How to write a generator function (the 1st parameter of fit_generator)?

• I only know the generator function aims to feed data batch by batch.

  As the name suggests, the .fit_generator function assumes there is an underlying function that is generating the data for it.

• What should be included in this geneator function? What should be returned? Any related example?

Mark: I have read several online examples (e.g., this and this). They use images as example, which is not my case (csv data only), and not easy to understand.

Answer 1

Ill supply two tutorials I used when I first started using fit_generator. That being said the first thing to remember is that a generator is essentially like any other function your write that returns something with the exception is that the function runs a continuous loop that is designed not to exit. For example, in a normal function, you would use return to return some chunk of data every time that function is called. In a generator, the function returns data in chunks continuously until there is no data to return. This is what the yield statement does. On my project, I have a couple of terabytes of signal data. Even on a powerful server that is a little bit too much data for hardware I use. So the generator function serves up chunks of data in batches, it can be run in parallel as well to increase speed. This approach also greatly reduces your memory usage and usually, your system's memory will hit some point and not really change until all data has been fed to the DNN. I apologize for the poor explanation but I am pretty new to this as well :-) I am sure someone else will provide a better answer but this should get you started.

Here is one good tutorial https://www.pyimagesearch.com/2018/12/24/how-to-use-keras-fit-and-fit_generator-a-hands-on-tutorial/

and a very short one https://medium.com/@fromtheast/implement-fit-generator-in-keras-61aa2786ce98

also, look at the docs https://keras.io/models/sequential/

and finally here is one I use on a current project. It is not very pretty but it works well

def batch_generator(X_train, Y_train):  

    while True:
         # samples_counter = 0
        for fl, lb in zip(X_train, Y_train):
            sam, lam = get_IQsamples(fl, lb)
            max_iter = sam.shape[0]
            sample = []  # store all the generated data batches
            label = [] # store all the generated label batches
            i = 0
            for d, l in zip(sam, lam):
                sample.append(d)
                label.append(l)
                i += 1
                if i == max_iter:
                    break
            sample = np.asarray(sample)        
            label = np.asarray(label)
            yield sample, label

Answer 2

On my phone right now but you should set the batch size to a specific size. The generator will keep passing batches back. Let's say your batch size is 32 but your almost out of sample and only 20 are left. The generator will pass back a batch that's partially full and that is fine. Someone else will have to answer the question on having each batch different but I cannot think what you would want that.