New answers tagged backpropagation
2
Of course that all weights are the same, but the update applied to the weights has a contribution from each of the timesteps, and the contribution associated with the first timesteps is what is more affected by the vanishing gradient problem.
1
According to Kingma and Ba (2014) Adam has been developed for "large datasets and/or high-dimensional parameter spaces".
The authors claim that: "[Adam] combines the advantages of [...] AdaGrad to deal with sparse gradients, and the ability of RMSProp to deal with non-stationary objectives" (page 9).
In the paper, there are some ...
0
Different operations on different elements don't prevent differentiation in any way.
Lets, say we call your above Loss function:
$$\mathcal L=L_1(\mathbf w) + L_2(\mathbf w)$$,
where $\mathbf w$ represents the weights of your model, $L_1$ and $L_2$ are the two loss functions you have defined using the different outputs of your model. The key point is that ...
0
Using momentum is a noise reduction (noisy gradients) and signal amplification strategy.
Imagine a large hill with a rough terrain with lots of ups-and-downs. We are trying to navigate to the bottom of the hill by using purely local information. A bad strategy is course correct frequently every time we see a potential new direction with steeper descent.
The ...
Top 50 recent answers are included
