Logistic regression, based on the logistic function $\sigma(x) = \frac{1}{1 + \exp(-x)}$, can be seen as a hypothesis testing problem. Where the reference distribution is the standard Logistic distribution where the p.m.f is

$f(x) = \frac{\exp(-x)}{[1 + \exp(-x)]^2}$

and the c.d.f is

$F(x) = \sigma(x) = \frac{1}{1 + \exp(-x)}$

The hypothesis to test is

$H_0: x \text{ isn't positive} \hspace{2.0cm} H_1: x \text{ is positive}$

The test statistic is $F(x)$. We reject $H_0$ if $F(x) \geq \alpha$ where $\alpha$ is the level of significance (in terms of hypothesis testing) or classification threshold (in terms of classification problem)

My question is that why they don't come up with the Standard normal distribution, which truly reflects the "distribution of nature", instead of Logistic distribution ?


Nice comparison.

Generally, we are allowed to experiment with as many distributions as we want, and find the one that suits our purpose. However, the normality assumption leads to an intractable derivation consisting of the notorious erf function.

Let's first pinpoint what is $x$ in the context of logistic regression. Logistic regression model can be written as: $$P(y=1|\boldsymbol{x})=\frac{1}{1+e^{-\boldsymbol{w}^t\boldsymbol{x}}}=F(\boldsymbol{w}^t\boldsymbol{x})$$ So your $x$ is actually $z=\boldsymbol{w}^t\boldsymbol{x}$. This means, although it is reasonable to assume that predicate $\boldsymbol{x}$ comes from a normal distribution, the same argument does not hold for a linear combination of its dimensions, i.e. $z$. In other words, the normal assumption is not as natural for $z$ as for $\boldsymbol{x}$.

But still, let's see what happens with normal assumption. The problem that we face here is analytical intractability. More specifically, to fit a similar model to observations using Maximum Likelihood, we need (1) derivative of cumulative distribution function (CDF) with respect to each parameter $w_i$, and (2) value of CDF for a given $z$ (see this lecture section 12.2.1 for more details).

For logistic distribution, the required gradient would be: $$\begin{align*} \frac{\partial F(\boldsymbol{x};\boldsymbol{w})}{\partial w_i}&=\frac{\partial (1+e^{-\boldsymbol{w}^t\boldsymbol{x}})^{-1}}{\partial w_i}= x_i e^{-\boldsymbol{w}^t\boldsymbol{x}}(1+e^{-\boldsymbol{w}^t\boldsymbol{x}})^{-2} =x_if(\boldsymbol{x};\boldsymbol{w}) \end{align*}$$

However for normal distribution, CDF is the erf function which does not have an exact formula, though, its gradient is tractable. Assuming $z \sim \mathcal{N}(0, 1)$, the gradient would be: $$\begin{align*} \frac{\partial F(\boldsymbol{x};\boldsymbol{w})}{\partial w_i}&=\frac{\partial \left(\frac{1}{2}+\frac{1}{2}\text{erf}\left(\frac{z}{\sqrt{2}}\right)\right)}{\partial w_i}=\frac{x_i}{\sqrt{2 \pi}} e^{-\frac{(\boldsymbol{w}^t\boldsymbol{x})^2}{2}}=x_if(\boldsymbol{x};\boldsymbol{w}) \end{align*}$$

In summary, the normality assumption is not as justified for $z=\boldsymbol{w}^t\boldsymbol{x}$ as for $\boldsymbol{x}$, and it leads to an intractable CDF. Therefore, we continue using the good old logistic regression!

Here is a visual comparison of normal and logistic CDFs:

taken from a post by Enrique Pinzon, which implies a large analytical cost for a small difference!


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.