5
$\begingroup$

I understand the softmax equation is

$\boldsymbol{P}(y=j \mid x)=\frac{e^{x_{j}}}{\sum_{k=1}^{K} e^{x_{k}}}$

My question is: why use $e^x$ instead of say, $3^x$. I understand $e^x$ is it's own derivative, but how is that advantageous in this situation?

I'm generally trying to understand why euler's number appears everywhere, especially in statistics and probability, but specifically in this case.

$\endgroup$

1 Answer 1

7
$\begingroup$

Choosing a different base would squash the graph of the function uniformly in the horizontal direction, since $$ a^x = e^{x\cdot \ln(a)}. $$

The exponential function with base $e$ is widely considered the simplest exponential function. It has nice properties that no other base has, mainly:

  • The function $e^x$ is its derivative.
  • It has a particularly simple power series expansion: $$ e^x = 1 + x + \frac12 x^2 + \frac16 x^3 + \cdots + \frac1{n!}x^n + \cdots $$ All of the coefficients are rational numbers. If the base had been something intuitively "nicer" than $e$, such as an integer, the coefficients would need to be irrational.

For this reason, most mathematicians will pick $e^x$ when they need an exponential function and have no particular reason to choose one base over another. (Except for computer scientists and information theorists, who sometimes prefer $2^x$).

$\endgroup$
4
  • $\begingroup$ Reference: "hmakholm left over Monica (math.stackexchange.com/users/14366/hmakholm-left-over-monica), Why does sigmoid function use e instead of another constant?, URL (version: 2019-04-21): math.stackexchange.com/q/3195726 " $\endgroup$ Aug 26, 2022 at 21:01
  • 1
    $\begingroup$ It also has nice properties for complex numbers. $\endgroup$ Aug 27, 2022 at 9:53
  • $\begingroup$ Did this answer the question? Do those properties you've listed make a difference over using 3^x (e.g. does "rational coefficients" make it run quicker on a GPU, or anything like that?). If not, could we consider that 2.6^x vs. e^x vs. 2.9^x is just another hyperparameter of our models, that we could be tuning for? $\endgroup$ Aug 29, 2022 at 20:18
  • $\begingroup$ I don't think there's a need to see it as a hyperparameter. This is for a simple reason: the prior layers are multiplied by trainable weights and change during back-propagation. This is relevant to your question because the output value of the prior neurons are the inputs to the next layer. In essence, using the property $a^x = e^{x\cdot \ln(a)}$ you can think of the $ln(a)$ as the constant multiplied by the prior layer weight, which is adjusted again in Back Propagation; there's no need for hyperparameter tuning. $\endgroup$ Aug 29, 2022 at 20:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.