1
$\begingroup$

The constraint that the n outputs must sum to $1$ means that only $n−1$ parameters are necessary; the probability of the $n^{th}$ value may be obtained by subtracting the first $n−1$ probabilities from $1$. We can thus impose a requirement that one element of $z$ be fixed. For example, we can require that $z_n = 0$. Indeed, this is exactly what the sigmoid unit does.

  1. What does it mean by "impose a requirement that one element of $z$ be fixed." I understand you can get the $n^{th}$ one if you know n-1 but "fixed"?

Defining $P (y = 1 | x ) = \sigma(z )$ is equivalent to defining $P (y = 1 | x) = softmax(z)_1$ with a two-dimensional $z$ and $z_1 = 0$. Both the $n − 1$ argument and the $n$ argument approaches to the softmax can describe the same set of probability distributions, but have different learning dynamics.

  1. How is it equivalent to having $z_1=0$?
$\endgroup$
1
$\begingroup$

The book quotes in your question are the outline of a proof that shows single output sigmoid represents the same class probabilities as a two dimensional softmax.

  1. What does it mean by "impose a requirement that one element of $z$ be fixed." I understand you can get the $n^{th}$ one if you know n-1 but "fixed"?

It doesn't mean the value has to be fixed. It means that we can choose any $z_n$ value, and have the other's adjusted to represent any valid target probabilities. So for argument's sake, we can pick any fixed value.

  1. How is it equivalent to having $z_1=0$?

It is an equivalence between $\sigma(z)$ and $softmax(z)_1$ with $z_1 = 0$:

$\sigma(z) = \frac{1}{1 + e^{-z}}$

For softmax, this is the function for the first element from a 2-dimensional output:

$softmax(z)_1 = \frac{e^{z_1}}{\sum_{i=1}^{2}e^{z_i}} = \frac{e^{z_1}}{e^{z_1} + e^{z_2}}$

But we have set $z_1 = 0$, and $e^0 = 1$ so:

$softmax(z)_1 = \frac{1}{1 + e^{z_2}}$

This is the same function as $\sigma(-z_2)$, so the two functions have identical form (if you plot them, they overlap), although the parameters drive the specific value differently.

Note that as per step 1, we have not lost anything about the general behaviour by setting $z_1 = 0$, we just did it to simplify the equation. In fact it is possible to make a very similar argument without fixing any $z$ values, and you end up with a more generic linear combination of $z_1$ and $z_2$ in the softmax function being equivalent to just $z$ in $\sigma(z)$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.