0
$\begingroup$

In AI libraries such as, Tensorflow, Keras etc., how the big output numbers are dealt during training process? For example, for classification layer, it will have some outputs such as [23.4, -21254.3, 32123.4]. In next step, these numbers will go into softmax function which will take power of base e with each output numbers. However, this may result in extreme big numbers in case of extremely large positive number or extremely small negative number. So, how are extreme cases dealt during training process?

$\endgroup$
1
  • $\begingroup$ Is this happening even when you have BatchNormalization layer? $\endgroup$
    – 10xAI
    Feb 9 at 13:46
1
$\begingroup$

As you stated in your question, those numbers go into a softmax function. Another name for softmax is normalized exponential function. Softmax normalizes numbers where the sum is constrained to be 1 and each value becomes the probability of categorical membership.

In the specific case of [23.4, -21254.3, 32123.4], applying the softmax transforms them to [0, 0, 1]. The numbers become either 0 or 1 because of numerical underflow.

$\endgroup$
2
  • $\begingroup$ Thank you for the answer. $\endgroup$ Feb 18 at 7:15
  • $\begingroup$ However, I have one confusion that in case of transforming numbers to [0,0,1] in case of underflow, is there any specific rule? $\endgroup$ Feb 18 at 7:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.