5
$\begingroup$

The sigmoid function's derivative is of that form, and so is the softmax function's. Is this by design, or some strange coincidence that seems to work for ML models/neural networks?

$\endgroup$
10
$\begingroup$

Sigmoid function is a partial case of softmax, when the number of classes $K=2$. That's why the similarity of their derivatives shouldn't surprise you.

Why do so many functions used in data science have derivatives of the form f(x)*(1-f(x))?

If you consider the following differential equation

$y' = y \cdot (1-y)$

you will find the general solution in the form

$y(x) = \frac{e^x}{c + e^x} = \frac{1}{1 + c e^{-x}}$

So, in some sense, there are not that many functions with this property: they are "very like" sigmoid function.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.