Often we find that the NN training process could be highly constrained by GPU memory size, like in object detection models the training batch size could be limited to 1 or 2, which has inflicted those working in such tasks for long and led to a number of tricks proposed sorely for the work-arounds on such limited memory/batch size scenarios. So why is the GPU memory limited to, e.g. 16GB on common P100/V100, compared to up to 256GB RAM for CPU on high performance workstations? What is the obstacle for Nvidia to release a, say, 128GB V100? And what would be the cost for that?
Basically, the reason is because the memory of the GPU is much more expensive. This is due to the fact that GPUs using a very high memory bandwidth. A normal GTX 480 has 384 bit wide bus, which allows a maximum transfer rate of ~180 GB/sec. However, an Intel i7 CPU using a 192 bit wide bus has a maximum of about ~30 GB/sec depending of what type of RAM you use. This means that the transfer of data in the GPU is about 5x faster compared to a CPU. Therefore, memory for GPUs is much more expensive but you can get more transfer ratio.
Additionally increasing more chips, the subsequent increase in capacity could end up messing with the timing of transfer of data.
The simple answer is that it costs more to produce X amount of GPU memory compared to an X amount of CPU memory.
Architecturally, the CPU is composed of just few cores with lots of cache memory that can handle a few software threads at a time. In contrast, a GPU is composed of hundreds of cores that can handle thousands of threads simultaneously.