New answers tagged

2

Of course that all weights are the same, but the update applied to the weights has a contribution from each of the timesteps, and the contribution associated with the first timesteps is what is more affected by the vanishing gradient problem.


1

According to Kingma and Ba (2014) Adam has been developed for "large datasets and/or high-dimensional parameter spaces". The authors claim that: "[Adam] combines the advantages of [...] AdaGrad to deal with sparse gradients, and the ability of RMSProp to deal with non-stationary objectives" (page 9). In the paper, there are some ...


0

Different operations on different elements don't prevent differentiation in any way. Lets, say we call your above Loss function: $$\mathcal L=L_1(\mathbf w) + L_2(\mathbf w)$$, where $\mathbf w$ represents the weights of your model, $L_1$ and $L_2$ are the two loss functions you have defined using the different outputs of your model. The key point is that ...


0

Using momentum is a noise reduction (noisy gradients) and signal amplification strategy. Imagine a large hill with a rough terrain with lots of ups-and-downs. We are trying to navigate to the bottom of the hill by using purely local information. A bad strategy is course correct frequently every time we see a potential new direction with steeper descent. The ...


Top 50 recent answers are included