I trained a model and froze it into a PB (protocol buffer) file and a directory of some variables, and the total size is about 31M. We deployed it using a GPU card and followed this answer and set the per_process_gpu_memory_fraction to a very little number to make the memory to be about 40M. The program performs very well but when we check the GPU usage by nvidia-smi which shows that the memory usage is about 500M.

Then my question is how can I justify this gap? How can we reduce that? Can we do something like quantization to decrease the 500M? We want to deploy it into an edge device so the 500M is too large.


I cannot answer regarding the gap. What I can help you is how to reduce the memory usage.

One thing you can try is you can convert your weights to fp16 and do inference with fp16. Don't worry you would not lose significant numerical precision especially since you are using that for deployment. In fact, this is how people do forward pass on mixed precision training. Theoretically this could increase speed and reduce memory usage. Hope it helps.

  • $\begingroup$ Do you mean reducing the 31M to 16M or something like that? But the total memory would be 500M - 15M, actually no decrease. $\endgroup$ – Lerner Zhang Nov 9 '19 at 4:17
  • $\begingroup$ Theoretically, yes, so using fp16 means your input and weights are all calculated with fp16 which costs half of memory of default fp32. In practice though im not sure how much it would cut from 500M, but I am sure the usage should be less than that. As I have mentioned earlier I am not entirely sure why it could grow beyond your threshold of 40M. $\endgroup$ – Yohanes Alfredo Nov 9 '19 at 7:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.