0
$\begingroup$

Let's say I'm performing Stochastic Gradient Descent (SGD) on binary cross entropy error while optimizing weight $w_{2}$.

Binary cross entropy error:

$$L(y|p(x_{i}))=-y_{i}*ln(p(x_{i}))-(1-y_{i})*ln(1-p(x_{i}))$$

The derivative of such roadmap would be:

$$\frac{\partial L(y|p(x_{i}))}{\partial w_{1}}$$

We can use a chain rule to solve this:

$$\frac{\partial L(y|p(x_{i}))}{\partial w_{1}}=\frac{\partial L(y|p(x_{i}))}{p_{i}}*\frac{\partial p_i}{w_1}$$

First term is easily understandable, it's just a differentiation of a loss function with respect to the estimated probability:

$$\frac{y_i -p_i}{p_i(1 - p_i)}$$

But the second term seems a little weird, from my understanding $p_i$ is just estimation of Bernoulli distribution (therefore it should be Empirical distribution). Somehow it's derivative is:

$$p_{i}*(1-p_i)*x_i$$

This is definitely somehow associated with the Bernoulli distribution, but how exactly? What is the definition of $p_i$ in this particular case?

Thank you!

$\endgroup$

1 Answer 1

2
$\begingroup$

Here the function used for $p_{i}$ is sigmoid function.
So,
$p_{i}$ = $\frac{1}{1+e^{-\Sigma w_{j} ∗ x_{i}^j}}$
$\frac{∂p_{i}} {w_{j}}$= $\frac{-1}{(1+e^{-\Sigma w_{j} ∗ x_{i}^j})^2}$ ∗ $e^{-\Sigma w_{j} ∗ x_{i}^j}$ ∗ $(-x_{i}^j)$

$\frac{∂p_{i}} {w_{j}}$ = $\frac{1}{(1+e^{-\Sigma w_{j} ∗ x_{i}^j})}$ ∗ $\frac{e^{-\Sigma w_{j} ∗ x_{i}^j}}{(1+e^{-\Sigma w_{j} ∗ x_{i}^j})}$ ∗ $x_{i}^j$

$1-p_{i}$ = $\frac{e^{-\Sigma w_{j} ∗ x_{i}^j}}{1+e^{-\Sigma w_{j} ∗ x_{i}^j}}$

$\frac{∂p_{i}} {w_{j}}$ = $p_{i}$ ∗ $(1-p_{i})$ ∗ $x_{i}^j$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.