Keras Sequence Generator leads to huge memory usage when used with fit_generator

Question

I want to use a fit_generator to stabilize the memory usage when training with very large datasets. Q1: From my understanding, the generator puts batches in a queue which is fetched by the fit_generator function of keras to train the model on that batch. After the raining with that batch, it should release that batch from memory?

Q2: In my example, the memory increases with every batch. No memory cleaning is seen. Furthermore the memory does switch into the swap memory. afterwards the memory stays stable but the swap memory increases until the program crashes.

Q3: I am using the cpu only for testing purposes right now. Does the fit_generator can work as intended when everything has to run on the ram of the computer and everything is computed with the cpu?

Here is my code:

class DataGenerator(Sequence):
    def __init__(self, batch_size, id_list, dim, shuffle=True):
        # batch_size: batch_size at each iteration
        self.batch_size = batch_size
        self.id_list = id_list
        self.dim = dim
        self.batch_size = batch_size
        self.shuffle = shuffle
        self.on_epoch_end()

    def __len__(self):
        # Denotes the number of batches per epoch
        return int(np.floor(len(self.id_list)/self.batch_size))

    def __getitem__(self, index):
        '''''
        MAIN function:
        - index tells the number of batches that have already been processed
        :return: X and y with shape [batch_size, window_size, features]        
        '''''
        X, y = self.__data_generation(self.id_list[index])
        return X, y


    @profile
    def on_epoch_end(self):
        if self.shuffle:
            random.shuffle(self.id_list)
        gc.collect()

    def __data_generation(self, fname):
        # 1. load data from specified path
        data = self.__load_data(filename=fname)
        data_x = np.squeeze(data[0, 0])
        data_y = np.squeeze(data[1, 0])
        return data_x, data_y

        # --------------------------------------- utility functions data loading

    def __load_data(self, filename):
        data = np.load(filename, allow_pickle=True)
        # load data from the filename
        return data

The generator is than put into the fit_generator function

train_gen = DataGenerator(batch_size=1,
                             id_list=train_files,
                             dim=(256, 6),
                             shuffle=True)

model.fit_generator(generator=train_gen,
                       epochs=2,
                       verbose=1,
                       use_multiprocessing=True,
                       workers=5,
                       max_queue_size=10)

I would like to know if my concept is wrong or if I misunderstood the use of the fit_generator function.

Answer 1

Q1: Yes, in theory memory should be released Q2: This is not a question Q3: Yes, fit_generator can work perfectly fine on a CPU.

Your data generator looks fine as far as I can see, based on my own experiences with implementing them, so I can only advise on how to proceed to find the root-cause for now as a potential remedy for the actual problem.

The first thing I would try is to set multi-processing to False and workers to 1. I have had some issues myself a while back when using the multi-processing functionality, and while I do not remember if it was memory related or race-condition related, I would start by checking if things work fine. The second thing I would look at is alternative ways of loading the files and see if these things still lead to memory leaks.