1
$\begingroup$

I want to use a fit_generator to stabilize the memory usage when training with very large datasets. Q1: From my understanding, the generator puts batches in a queue which is fetched by the fit_generator function of keras to train the model on that batch. After the raining with that batch, it should release that batch from memory?

Q2: In my example, the memory increases with every batch. No memory cleaning is seen. Furthermore the memory does switch into the swap memory. afterwards the memory stays stable but the swap memory increases until the program crashes.

Q3: I am using the cpu only for testing purposes right now. Does the fit_generator can work as intended when everything has to run on the ram of the computer and everything is computed with the cpu?

Here is my code:

class DataGenerator(Sequence):
    def __init__(self, batch_size, id_list, dim, shuffle=True):
        # batch_size: batch_size at each iteration
        self.batch_size = batch_size
        self.id_list = id_list
        self.dim = dim
        self.batch_size = batch_size
        self.shuffle = shuffle
        self.on_epoch_end()

    def __len__(self):
        # Denotes the number of batches per epoch
        return int(np.floor(len(self.id_list)/self.batch_size))

    def __getitem__(self, index):
        '''''
        MAIN function:
        - index tells the number of batches that have already been processed
        :return: X and y with shape [batch_size, window_size, features]        
        '''''
        X, y = self.__data_generation(self.id_list[index])
        return X, y


    @profile
    def on_epoch_end(self):
        if self.shuffle:
            random.shuffle(self.id_list)
        gc.collect()

    def __data_generation(self, fname):
        # 1. load data from specified path
        data = self.__load_data(filename=fname)
        data_x = np.squeeze(data[0, 0])
        data_y = np.squeeze(data[1, 0])
        return data_x, data_y

        # --------------------------------------- utility functions data loading

    def __load_data(self, filename):
        data = np.load(filename, allow_pickle=True)
        # load data from the filename
        return data

The generator is than put into the fit_generator function

train_gen = DataGenerator(batch_size=1,
                             id_list=train_files,
                             dim=(256, 6),
                             shuffle=True)

model.fit_generator(generator=train_gen,
                       epochs=2,
                       verbose=1,
                       use_multiprocessing=True,
                       workers=5,
                       max_queue_size=10)

I would like to know if my concept is wrong or if I misunderstood the use of the fit_generator function.

$\endgroup$
1
$\begingroup$

Q1: Yes, in theory memory should be released Q2: This is not a question Q3: Yes, fit_generator can work perfectly fine on a CPU.

Your data generator looks fine as far as I can see, based on my own experiences with implementing them, so I can only advise on how to proceed to find the root-cause for now as a potential remedy for the actual problem.

The first thing I would try is to set multi-processing to False and workers to 1. I have had some issues myself a while back when using the multi-processing functionality, and while I do not remember if it was memory related or race-condition related, I would start by checking if things work fine. The second thing I would look at is alternative ways of loading the files and see if these things still lead to memory leaks.

|improve this answer|||||
$\endgroup$
  • $\begingroup$ Thanks for your thoughts. I have already read that numpy could be a potential memory issue when using it with the keras fit_generator. So I might try to change this. I already tried to go with multi_processing=False, workers=1 and max_q_size=0.Still the same error appears. It might could have something to do with cyclng references. But I don't see where this could occur. Any idea? $\endgroup$ – alex_v_1247 Dec 2 '19 at 7:08
  • $\begingroup$ I got it to work. To run it only on the cpu was the issue. As soon as I run it with gpu support, everything worked perfectly. $\endgroup$ – alex_v_1247 Dec 3 '19 at 13:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.