I need to train 5 models each for 10 times using tensorflow and keras for a homework. 2 of the models are multilayer perceptron models and remaining 3 of them are CNN models. I am training using my GPU, GTX1660Ti. I have no issue training MLP models, but when I try to train the CNN models I get the error below:

enter image description here

I have added these two lines of code to solve this issue:

physical_devices = tf.config.list_physical_devices('GPU')
tf.config.experimental.set_memory_growth(physical_devices[0], True)

However, when I try to train all of the models in a single script, my process gets killed after succesfully training 3 models:

enter image description here

Training dataset contains 27000 images, test dataset contains 5000 images, and validation dataset contains 3000 images each 28*28. I suspect gpu memory is the issue but I do not know the details. Any help is appreciated.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.