0
$\begingroup$

I saw this equation in somebody's code which is an alternative approach to implementing the softmax in order to avoid underflow by division by large numbers.

softmax = e^(matrix - logaddexp(matrix)) = E^matrix / sumexp(matrix)

logsumexp = scipy.special.logsumexp(matrix, axis=-1, keepdims=True)
softmax = np.exp(matrix - logsumexp)

I understand that when you log equations that use division you would then subtract, i.e. log(1/2) = log(1) - log(2). However, in the implantation of the code above, shouldn't they also log the matrix in order to subract the logsumexp?

$\endgroup$

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.