Tag Info

2

If the function is convex, then all local minimum are global minimum. If $4-\beta^2 >0$, then the function is convex and hence the local minimum is indeed the global minimum. If $\beta = \pm 2$, then we have $f(x,y)=(x\pm y)^2+x+2y$, $f(x, \mp x)=x\mp2x$ of which we can make it arbitrary large or small. If $4-\beta^2 < 0$, then it is indefinite, the ...

1

You can take a look at dropout, it will get you out of the local minima you are stuck in.

1

tl;dr: You should project the gradients before feeding them to Adam. You should also do the clipping afterwards in case of non-linear constraints and to avoid accumulation of numerical errors. First, assuming normal SGD, for the specific case of (approximately) linear equality constraints, this can be done by projecting the gradient to the tangent space ...

1

According to Kingma and Ba (2014) Adam has been developed for "large datasets and/or high-dimensional parameter spaces". The authors claim that: "[Adam] combines the advantages of [...] AdaGrad to deal with sparse gradients, and the ability of RMSProp to deal with non-stationary objectives" (page 9). In the paper, there are some ...

Top 50 recent answers are included