I am trying to understand gradient calculation for skip gram with softmax output and cross entropy loss.

I am referring these articles: 1, 2, 3.

The all calculate the error as follows:

$$E=-\sum_{c=1}^Cu_{j_{c^*}}+C.log\left(\sum_{j'=1}^v e^uj'\right)$$

Step 1: Then they calculate the gradient of Error $E$ with respect to $u_j$ as follows:

$$\frac{\partial E}{\partial u_j}=-\sum_{c=1}^Cu_{j_{c^*}}+C.\frac{1}{\sum_{j'=1}^v e^{u_j^*}}\frac{\partial }{\partial u_j}\left(\sum_{j'=1}^v e^uj'\right)$$

Q1. Why the first term above $-\sum_{c=1}^Cu_{j_{c^*}}$ left untouched? That is, why they have not taken its gradient with respect to $u_j$?

Step2: In next step, they do:

$$\frac{\partial E}{\partial u_j}=-\sum_{c=1}^C 1+\sum_{j=1}^V y_j$$

Q2. I am unable to get how both terms in above sum are obtained.

Step 3: Finally the articles do:

$$\frac{\partial E}{\partial u_j}=y_j-t_j$$

Q3. Again I did not get how this step is obtained.

Can someone please tell me what I am missing here?



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.