1
$\begingroup$

I was playing around with the softmax function and tried around with the numerical stability of softmax. If we increase the exponent in the numerator and denominator with the same value, the output of the softmax stays constant (see picture below where -Smax is added). I cannot figure out how to prove this numerical stability (although I read that it is true). Can anyone help me with the proof?

enter image description here

Improve this question
$\endgroup$

1 Answer 1

Reset to default
4
$\begingroup$

Take into accout that $e^{a-b} = e^a \cdot e^{-b}$, therefore:

$\dfrac{e^{s_{y_i} - s_{max}}}{\sum e^{s_k - s_{max}}} = \dfrac{e^{s_{y_i}} e^{- s_{max}}}{\sum e^{s_k} e^{- s_{max}}} = \dfrac{e^{s_{y_i}} e^{- s_{max}}}{e^{- s_{max}} \sum e^{s_k}} = \dfrac{e^{s_{y_i}}}{\sum e^{s_k}} $

Improve this answer
$\endgroup$
3
  • $\begingroup$ Now it is clear, thank you! $\endgroup$
    – Nicoinlas
    Feb 2, 2022 at 19:32
  • $\begingroup$ Please consider marking the answer as correct with the tick mark. $\endgroup$
    – noe
    Feb 2, 2022 at 20:12
  • $\begingroup$ @Nicoinlas welcome to DSSE; please see What should I do when someone answers my question? $\endgroup$
    – desertnaut
    Feb 2, 2022 at 21:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.