Differentiating roadmap of a loss function

Question

Let's say I'm performing Stochastic Gradient Descent (SGD) on binary cross entropy error while optimizing weight $w_{2}$.

Binary cross entropy error:

$$L(y|p(x_{i}))=-y_{i}*ln(p(x_{i}))-(1-y_{i})*ln(1-p(x_{i}))$$

The derivative of such roadmap would be:

$$\frac{\partial L(y|p(x_{i}))}{\partial w_{1}}$$

We can use a chain rule to solve this:

$$\frac{\partial L(y|p(x_{i}))}{\partial w_{1}}=\frac{\partial L(y|p(x_{i}))}{p_{i}}*\frac{\partial p_i}{w_1}$$

First term is easily understandable, it's just a differentiation of a loss function with respect to the estimated probability:

$$\frac{y_i -p_i}{p_i(1 - p_i)}$$

But the second term seems a little weird, from my understanding $p_i$ is just estimation of Bernoulli distribution (therefore it should be Empirical distribution). Somehow it's derivative is:

$$p_{i}*(1-p_i)*x_i$$

This is definitely somehow associated with the Bernoulli distribution, but how exactly? What is the definition of $p_i$ in this particular case?

Thank you!

Answer 1

Here the function used for $p_{i}$ is sigmoid function.
So,
$p_{i}$ = $\frac{1}{1+e^{-\Sigma w_{j} ∗ x_{i}^j}}$
$\frac{∂p_{i}} {w_{j}}$= $\frac{-1}{(1+e^{-\Sigma w_{j} ∗ x_{i}^j})^2}$ ∗ $e^{-\Sigma w_{j} ∗ x_{i}^j}$ ∗ $(-x_{i}^j)$

$\frac{∂p_{i}} {w_{j}}$ = $\frac{1}{(1+e^{-\Sigma w_{j} ∗ x_{i}^j})}$ ∗ $\frac{e^{-\Sigma w_{j} ∗ x_{i}^j}}{(1+e^{-\Sigma w_{j} ∗ x_{i}^j})}$ ∗ $x_{i}^j$

$1-p_{i}$ = $\frac{e^{-\Sigma w_{j} ∗ x_{i}^j}}{1+e^{-\Sigma w_{j} ∗ x_{i}^j}}$

$\frac{∂p_{i}} {w_{j}}$ = $p_{i}$ ∗ $(1-p_{i})$ ∗ $x_{i}^j$