Timeline for What does the one function $\mathbf{1}_{i,y^{(t)}}$ exactly mean in backward propagation of RNN in the book "Deep learning" of Bengio
Current License: CC BY-SA 4.0
8 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Sep 25, 2018 at 1:33 | comment | added | gph | yes, in the simple example it is a scalar, then it is generalized to vector version. | |
| Sep 21, 2018 at 15:20 | comment | added | user12075 | you are right. I think the author means $t^{(t)}$ to be a scalar. I was thinking it as a one hot encoding, but it's not here. | |
| Sep 21, 2018 at 7:34 | comment | added | gph | Thank you for help! I was too impatient to ask the question when I hadn't read the chapters before RNN because I had thought it simple. I read the underlying chapter again and wrote an answer below to prove the equation (10.18). By the way, I think $y^{(t)}$ here is a scalar at certain time t. | |
| Sep 20, 2018 at 3:37 | comment | added | user12075 | $L$ is the cross-entropy loss defined in (10.14). For the indicator function, you are right that the condition is "scalar $i$"=="vector $y^{t}$", in this case the indicator function returns a vector whose k'th elements are indicator of whether that element in $i$ equals the k'th element of $y^{t}$. Similar like you compare a 1-D numpy array with a scalar in python. | |
| Sep 20, 2018 at 3:16 | comment | added | gph | And the expression of $L$? I have thought it was based on assumption that :$p(y|x;\theta)\sim \mathcal{N}(y,\mu=\hat{y},\sigma=1)$, i.e Gaussian distribution. So $L=\sum_t\frac{1}{2}\Vert\hat{\boldsymbol{y}}^{(t)}-\boldsymbol{y}^{(t)}\Vert^2+Constant$ But it seems that here the authors use the cross entropy loss like $L=\sum_t \boldsymbol{y}^{(t)}t\log{\hat{\boldsymbol{y}}^{(t)}}$? So do you know what is the expression of loss $L$ here?Thank you! | |
| Sep 20, 2018 at 2:59 | comment | added | gph | Thank you for your help! I have a further question that shouldn't $y^{(t)}$ be a vector $\boldsymbol{y}^{t}$ with the same length as $\hat{\boldsymbol{y}}^{(t)}$? Then how could it be equal to a scalar $i$? | |
| Sep 20, 2018 at 2:54 | vote | accept | gph | ||
| Sep 19, 2018 at 15:23 | history | answered | user12075 | CC BY-SA 4.0 |