Let's say I'm performing Stochastic Gradient Descent (SGD) on binary cross entropy error while optimizing weight $w_{2}$.
Binary cross entropy error:
$$L(y|p(x_{i}))=-y_{i}*ln(p(x_{i}))-(1-y_{i})*ln(1-p(x_{i}))$$
The derivative of such roadmap would be:
$$\frac{\partial L(y|p(x_{i}))}{\partial w_{1}}$$
We can use a chain rule to solve this:
$$\frac{\partial L(y|p(x_{i}))}{\partial w_{1}}=\frac{\partial L(y|p(x_{i}))}{p_{i}}*\frac{\partial p_i}{w_1}$$
First term is easily understandable, it's just a differentiation of a loss function with respect to the estimated probability:
$$\frac{y_i -p_i}{p_i(1 - p_i)}$$
But the second term seems a little weird, from my understanding $p_i$ is just estimation of Bernoulli distribution (therefore it should be Empirical distribution). Somehow it's derivative is:
$$p_{i}*(1-p_i)*x_i$$
This is definitely somehow associated with the Bernoulli distribution, but how exactly? What is the definition of $p_i$ in this particular case?
Thank you!