New answers tagged


Of course that all weights are the same, but the update applied to the weights has a contribution from each of the timesteps, and the contribution associated with the first timesteps is what is more affected by the vanishing gradient problem.


According to Kingma and Ba (2014) Adam has been developed for "large datasets and/or high-dimensional parameter spaces". The authors claim that: "[Adam] combines the advantages of [...] AdaGrad to deal with sparse gradients, and the ability of RMSProp to deal with non-stationary objectives" (page 9). In the paper, there are some ...


Different operations on different elements don't prevent differentiation in any way. Lets, say we call your above Loss function: $$\mathcal L=L_1(\mathbf w) + L_2(\mathbf w)$$, where $\mathbf w$ represents the weights of your model, $L_1$ and $L_2$ are the two loss functions you have defined using the different outputs of your model. The key point is that ...


Using momentum is a noise reduction (noisy gradients) and signal amplification strategy. Imagine a large hill with a rough terrain with lots of ups-and-downs. We are trying to navigate to the bottom of the hill by using purely local information. A bad strategy is course correct frequently every time we see a potential new direction with steeper descent. The ...

Top 50 recent answers are included