I saw this equation in somebody's code which is an alternative approach to implementing the softmax in order to avoid underflow by division by large numbers.

softmax = e^(matrix - logaddexp(matrix)) = E^matrix / sumexp(matrix)

logsumexp = scipy.special.logsumexp(matrix, axis=-1, keepdims=True)
softmax = np.exp(matrix - logsumexp)

I understand that when you log equations that use division you would then subtract, i.e. log(1/2) = log(1) - log(2). However, in the implantation of the code above, shouldn't they also log the matrix in order to subract the logsumexp?



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.