0
$\begingroup$

I have a large network that is somewhat similar to Wavenet. Although (it seems) that my GPU has enough memory, I get an out of memory error on fitting (see logs below).

Any idea?

How can I troubleshoot these kind of CUDA driver issues?

2017-12-22 23:32:05.288986: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorf low\core\common_runtime\gpu\gpu_device.cc:955] Found device 0 with properties: name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate (GHz) 1.6575 pciBusID 0000:01:00.0 Total memory: 11.00GiB Free memory: 10.71GiB

2017-12-22 23:32:05.288986: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorf low\core\common_runtime\gpu\gpu_device.cc:976] DMA: 0 2017-12-22 23:32:05.288986: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorf low\core\common_runtime\gpu\gpu_device.cc:986] 0: Y 2017-12-22 23:32:05.429386: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorf low\core\common_runtime\gpu\gpu_device.cc:1045] Creating TensorFlow device (/gpu:0) -> (device : 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:01:00.0)

2017-12-22 23:32:06.131386: E C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorf low\stream_executor\cuda\cuda_driver.cc:924] failed to allocate 10.17G (10922166272 bytes) fro m device: CUDA_ERROR_OUT_OF_MEMORY 2017-12-22 23:32:06.599386: E C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorf low\stream_executor\cuda\cuda_driver.cc:924] failed to allocate 9.15G (9829949440 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY 2017-12-22 23:32:07.332586: E C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorf low\stream_executor\cuda\cuda_driver.cc:924] failed to allocate 8.24G (8846954496 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY built graph

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

Did you try to start your training with a smaller data set and that worked?

How about using generators to gradually input your data? For me that worked quite well (in Keras : model.fit_generator(...))

Improve this answer
$\endgroup$
4
  • $\begingroup$ Actually when I restarted training the error went away. My question is more general on how to approach such errors (debugging). I guess we need to use the NVidia CUDA profiler. $\endgroup$
    – Eran
    Dec 26, 2017 at 21:03
  • $\begingroup$ Did you have an other Model running in parallel and did not set the allow growth parameter (config = tf.ConfigProto() config.gpu_options.allow_growth=True sess = tf.Session(config=config))? Then it could be that the model before have allocated all the space. $\endgroup$
    – Mimi Müller
    Dec 26, 2017 at 21:38
  • $\begingroup$ I cannot find such API in Keras: fit_generator. Did you mean: medium.com/@fromtheast/… ? $\endgroup$
    – Eran
    Jan 24, 2018 at 11:07
  • $\begingroup$ You can find it at: keras.io/models/sequential just search for fit_generator. But yes, it is the same, he also refers to that page. $\endgroup$
    – Mimi Müller
    Feb 1, 2018 at 14:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.