The sigmoid function's derivative is of that form, and so is the softmax function's. Is this by design, or some strange coincidence that seems to work for ML models/neural networks?
$\begingroup$
$\endgroup$
Add a comment
|
$\begingroup$
$\endgroup$
Sigmoid function is a partial case of softmax, when the number of classes $K=2$. That's why the similarity of their derivatives shouldn't surprise you.
Why do so many functions used in data science have derivatives of the form f(x)*(1-f(x))?
If you consider the following differential equation
$y' = y \cdot (1-y)$
you will find the general solution in the form
$y(x) = \frac{e^x}{c + e^x} = \frac{1}{1 + c e^{-x}}$
So, in some sense, there are not that many functions with this property: they are "very like" sigmoid function.
