2
$\begingroup$

Often we find that the NN training process could be highly constrained by GPU memory size, like in object detection models the training batch size could be limited to 1 or 2, which has inflicted those working in such tasks for long and led to a number of tricks proposed sorely for the work-arounds on such limited memory/batch size scenarios. So why is the GPU memory limited to, e.g. 16GB on common P100/V100, compared to up to 256GB RAM for CPU on high performance workstations? What is the obstacle for Nvidia to release a, say, 128GB V100? And what would be the cost for that?

$\endgroup$
1
  • $\begingroup$ GPUs are SIMD computers, for each processing unit, there should be a separate memory. By increasing the number of memories, the processing units also should be increased. It costs a lot now. $\endgroup$
    – Media
    Dec 14 '18 at 7:48
1
$\begingroup$

Basically, the reason is because the memory of the GPU is much more expensive. This is due to the fact that GPUs using a very high memory bandwidth. A normal GTX 480 has 384 bit wide bus, which allows a maximum transfer rate of ~180 GB/sec. However, an Intel i7 CPU using a 192 bit wide bus has a maximum of about ~30 GB/sec depending of what type of RAM you use. This means that the transfer of data in the GPU is about 5x faster compared to a CPU. Therefore, memory for GPUs is much more expensive but you can get more transfer ratio.

Additionally increasing more chips, the subsequent increase in capacity could end up messing with the timing of transfer of data.

$\endgroup$
0
$\begingroup$

The simple answer is that it costs more to produce X amount of GPU memory compared to an X amount of CPU memory.

Architecturally, the CPU is composed of just few cores with lots of cache memory that can handle a few software threads at a time. In contrast, a GPU is composed of hundreds of cores that can handle thousands of threads simultaneously.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.