I want to use a fit_generator to stabilize the memory usage when training with very large datasets. Q1: From my understanding, the generator puts batches in a queue which is fetched by the fit_generator function of keras to train the model on that batch. After the raining with that batch, it should release that batch from memory?
Q2: In my example, the memory increases with every batch. No memory cleaning is seen. Furthermore the memory does switch into the swap memory. afterwards the memory stays stable but the swap memory increases until the program crashes.
Q3: I am using the cpu only for testing purposes right now. Does the fit_generator can work as intended when everything has to run on the ram of the computer and everything is computed with the cpu?
Here is my code:
class DataGenerator(Sequence): def __init__(self, batch_size, id_list, dim, shuffle=True): # batch_size: batch_size at each iteration self.batch_size = batch_size self.id_list = id_list self.dim = dim self.batch_size = batch_size self.shuffle = shuffle self.on_epoch_end() def __len__(self): # Denotes the number of batches per epoch return int(np.floor(len(self.id_list)/self.batch_size)) def __getitem__(self, index): ''''' MAIN function: - index tells the number of batches that have already been processed :return: X and y with shape [batch_size, window_size, features] ''''' X, y = self.__data_generation(self.id_list[index]) return X, y @profile def on_epoch_end(self): if self.shuffle: random.shuffle(self.id_list) gc.collect() def __data_generation(self, fname): # 1. load data from specified path data = self.__load_data(filename=fname) data_x = np.squeeze(data[0, 0]) data_y = np.squeeze(data[1, 0]) return data_x, data_y # --------------------------------------- utility functions data loading def __load_data(self, filename): data = np.load(filename, allow_pickle=True) # load data from the filename return data
The generator is than put into the fit_generator function
train_gen = DataGenerator(batch_size=1, id_list=train_files, dim=(256, 6), shuffle=True) model.fit_generator(generator=train_gen, epochs=2, verbose=1, use_multiprocessing=True, workers=5, max_queue_size=10)
I would like to know if my concept is wrong or if I misunderstood the use of the fit_generator function.