I need to train 5 models each for 10 times using tensorflow and keras for a homework. 2 of the models are multilayer perceptron models and remaining 3 of them are CNN models. I am training using my GPU, GTX1660Ti. I have no issue training MLP models, but when I try to train the CNN models I get the error below:
I have added these two lines of code to solve this issue:
physical_devices = tf.config.list_physical_devices('GPU') tf.config.experimental.set_memory_growth(physical_devices, True)
However, when I try to train all of the models in a single script, my process gets killed after succesfully training 3 models:
Training dataset contains 27000 images, test dataset contains 5000 images, and validation dataset contains 3000 images each 28*28. I suspect gpu memory is the issue but I do not know the details. Any help is appreciated.