I have a large network that is somewhat similar to Wavenet. Although (it seems) that my GPU has enough memory, I get an out of memory error on fitting (see logs below).
Any idea?
How can I troubleshoot these kind of CUDA driver issues?
2017-12-22 23:32:05.288986: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorf low\core\common_runtime\gpu\gpu_device.cc:955] Found device 0 with properties: name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate (GHz) 1.6575 pciBusID 0000:01:00.0 Total memory: 11.00GiB Free memory: 10.71GiB
2017-12-22 23:32:05.288986: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorf low\core\common_runtime\gpu\gpu_device.cc:976] DMA: 0 2017-12-22 23:32:05.288986: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorf low\core\common_runtime\gpu\gpu_device.cc:986] 0: Y 2017-12-22 23:32:05.429386: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorf low\core\common_runtime\gpu\gpu_device.cc:1045] Creating TensorFlow device (/gpu:0) -> (device : 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:01:00.0)
2017-12-22 23:32:06.131386: E C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorf low\stream_executor\cuda\cuda_driver.cc:924] failed to allocate 10.17G (10922166272 bytes) fro m device: CUDA_ERROR_OUT_OF_MEMORY 2017-12-22 23:32:06.599386: E C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorf low\stream_executor\cuda\cuda_driver.cc:924] failed to allocate 9.15G (9829949440 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY 2017-12-22 23:32:07.332586: E C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorf low\stream_executor\cuda\cuda_driver.cc:924] failed to allocate 8.24G (8846954496 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY built graph