0
$\begingroup$

My nvidia-smi shows the following:

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 418.87.01    Driver Version: 418.87.01    CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla V100-SXM2...  On   | 00000000:00:10.0 Off |                    0 |
| N/A   33C    P0    56W / 300W |  15323MiB / 16130MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla V100-SXM2...  On   | 00000000:00:11.0 Off |                    0 |
| N/A   33C    P0    57W / 300W |    462MiB / 16130MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   2  Tesla V100-SXM2...  On   | 00000000:00:12.0 Off |                    0 |
| N/A   55C    P0   266W / 300W |  15965MiB / 16130MiB |     99%      Default |
+-------------------------------+----------------------+----------------------+
|   3  Tesla V100-SXM2...  On   | 00000000:00:13.0 Off |                    0 |
| N/A   55C    P0   253W / 300W |  15965MiB / 16130MiB |     99%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
+-----------------------------------------------------------------------------+

In the first table, the memory usage is alot, for example, 15323MiB / 16130MiB. But in the 2nd table, it doesn't show any memory usage. Also, in the first table, the GPU 0 and 1 are not used at all, but why is the memory usage high?

I am asking this question because I am do a little training but it shows a Cuda out of memory error. But if I do the following command, it shows that memory availability is high:

> free -mh
              total        used        free      shared  buff/cache   available
Mem:           125G         26G         27G        2.1G         71G         96G
Swap:            0B          0B          0B
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.