0
$\begingroup$

What is it all about FP16, FP32 in Python? My potential Business Partner and I are building a Deep Learning Setup for working with time series. He came up with "FP16 and FP32" while finding a GPU. It looks like he's talking about Floating Point values in 16 vs 32bit. (Our data points look like this: "5989.12345", so I'm pretty sure 16bit ain't enough.)

Is FP16 a special technique GPUs use to improve performance or is it just a fancy term for using 16-bit float values instead of 32 standard floats?

$\endgroup$
  • 1
    $\begingroup$ Yes, FP16 means 16 bits in the context of GPU. $\endgroup$ – user3363813 Apr 27 at 21:17
3
$\begingroup$

FP32 and FP16 mean 32-bit floating point and 16-bit floating point. GPUs originally focused on FP32 because these are the calculations needed for 3D games. Nowadays a lot of GPUs have native support of FP16 to speed up the calculation of neural networks. If you look at some benchmarks (https://blog.slavv.com/titan-rtx-quality-time-with-the-top-turing-gpu-fe110232a28e) you will see that GPUs that support FP16 are almost twice as fast calculating FP16 than FP32. Taking into account that newer cards that support FP16 (like NVidia 2080 series) are also about 20% faster for FP32 compared to their predecessor (1080) you get an increase of 140% to train FP16 neural networks compared to FP32 on previous cards.

But there is a caveat. Your neural network needs to be written using FP16 and it should also have the same accuracy. FP16 have lower accuracy by design, because they have much less bits to represent the same number. Some choose to use mixed precision models in order to be fast and accurate (https://hackernoon.com/rtx-2080ti-vs-gtx-1080ti-fastai-mixed-precision-training-comparisons-on-cifar-100-761d8f615d7f). But you see in the last link that the speed is faster for mixed precision, but is not 2 times faster as when you use pure FP16. You could also theoretically use FP32 weights and convert some of them to FP16 weights, but the accuracy could fall.

So in the end you need to understand whether you could rewrite your neural network to use FP16 fully or partially. If you cannot then you do not get any additional benefits from FP16 compatible cards.

The maximum value for FP16 is 65504 and the minimum is 5.96 × 10−8. The best precision for 5989.12345 will most likely be 5988.0 (played with bits on https://www.h-schmidt.net/FloatConverter/IEEE754.html) If this precision and magnitude is not enough for you then you could scale your data before training to fit FP16 and then train with double the speed. Or use mixed precision models where you have FP32 as input and then reduce precision in later layers.

| improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ highly informational blog content, thx for that :) $\endgroup$ – Ishmael89 Apr 27 at 22:08
  • $\begingroup$ I also included a likely representation of your number in FP16. If you are satisfied with the answer it would be good if you could check the answer as the accepted answer and also upvote if you want. $\endgroup$ – keiv.fly Apr 27 at 22:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.