73

Is the learning rate related to the shape of the error gradient, as it dictates the rate of descent? In plain SGD, the answer is no. A global learning rate is used which is indifferent to the error gradient. However, the intuition you are getting at has inspired various modifications of the SGD update rule. If so, how do you use this information to inform ...


31

Gradient Descent is an algorithm which is designed to find the optimal points, but these optimal points are not necessarily global. And yes if it happens that it diverges from a local location it may converge to another optimal point but its probability is not too much. The reason is that the step size might be too large that prompts it recede one optimal ...


28

No. Gradient descent is used in optimization algorithms that use the gradient as the basis of its step movement. Adam, Adagrad, and RMSProp all use some form of gradient descent, however they do not make up every optimizer. Evolutionary algorithms such as Particle Swarm Optimization and Genetic Algorithms are inspired by natural phenomena do not use ...


24

In On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima there are a couple of intersting statements: It has been observed in practice that when using a larger batch there is a degradation in the quality of the model, as measured by its ability to generalize [...] large-batch methods tend to converge to sharp minimizers of ...


22

Below is a very good note (page 12) on learning rate in Neural Nets (Back Propagation) by Andrew Ng. You will find details relating to learning rate. http://web.stanford.edu/class/cs294a/sparseAutoencoder_2011new.pdf For your 4th point, you're right that normally one has to choose a "balanced" learning rate, that should neither overshoot nor converge too ...


21

When new observations are available, there are three ways to retrain your model: Online: each time a new observation is available, you use this single data point to further train your model (e.g. load your current model and further train it by doing backpropagation with that single observation). With this method, your model learns in a sequential manner and ...


16

Once a model is trained and you get new data which can be used for training, you can load the previous model and train onto it. For example, you can save your model as a .pickle file and load it and train further onto it when new data is available. Do note that for the model to predict correctly, the new training data should have a similar distribution as ...


16

I think in the original paper they suggest using $\log_2(N +1$), but either way the idea is the following: The number of randomly selected features can influence the generalization error in two ways: selecting many features increases the strength of the individual trees whereas reducing the number of features leads to a lower correlation among the trees ...


13

Let me give an explanation based on multivariate calculus. If you have taken a multivariate course, you will have heard that, given a critical point (point where the gradient is zero), the condition for this critical point to be a minimum is that the Hessian matrix is positive definite. As the Hessian is a symmetric matrix, we can diagonalize it. If we write ...


13

Asides from the points you mentioned (convergence to non-global minimums, and large step sizes possibly leading to non-convergent algorithms), "inflection ranges" might be a problem too. Consider the following "recliner chair" type of function. Obviously, this can be constructed so that there is a range in the middle where the gradient is the 0 vector. In ...


13

Here’s a blog post reviewing an article claiming SGD is a better generalized adapter than ADAM. https://shaoanlu.wordpress.com/2017/05/29/sgd-all-which-one-is-the-best-optimizer-dogs-vs-cats-toy-experiment/ There is often a value to using more than one method (an ensemble), because every method has a weakness.


11

There is no technique that will eliminate the risk of overfitting entirely. The methods you've listed are all just different ways of fitting a linear model. A linear model will have a global minimum, and that minimum shouldn't change regardless of the flavor of gradient descent that you're using (unless you're using regularization), so all of the methods you'...


10

Selecting a learning rate is an example of a "meta-problem" known as hyperparameter optimization. The best learning rate depends on the problem at hand, as well as on the architecture of the model being optimized, and even on the state of the model in the current optimization process! There are even software packages devoted to hyperparameter optimization ...


10

(My answer is based mostly on Adam: A Method for Stochastic Optimization (the original Adam paper) and on the implementation of rmsprop with momentum in Tensorflow (which is operator() of struct ApplyRMSProp), as rmsprop is unpublished - it was described in a lecture by Geoffrey Hinton .) Some Background Adam and rmsprop with momentum are both methods (...


9

You might find Chapter 8 of Deep Learning helpful. In it, the authors discuss training of neural network models. It's very intricate, so I'm not surprised you're having difficulties. One possibility (besides user error) is that your problem is highly ill-conditioned. Gradient descent methods use only the first derivative (gradient) information when ...


8

Copy-pasted from my masters thesis: If the loss does not decrease for several epochs, the learning rate might be too low. The optimization process might also be stuck in a local minimum. Loss being NAN might be due to too high learning rates. Another reason is division by zero or taking the logarithm of zero. Weight update tracking: Andrej Karpathy proposed ...


8

According to the title: No. Only specific types of optimizers are based on Gradient Descent. A straightforward counterexample is when optimization is over a discrete space where gradient is undefined. According to the body: Yes. Adam, Adagrad, RMSProp and other similar optimizers (Nesterov, Nadam, etc.) are all trying to propose an adaptive step size (...


7

This is simply trying to convey my intuition, i.e. no rigor. The thing with saddle points is that they are a type of optimum which combines a combination of minima and maxima. Because the number of dimensions are so large with deep learning, the probability that an optimum only consists of a combination of minima is very low. This means 'getting stuck' in a ...


7

Reinforcement learning is more about interacting with an environment, and while this could be posed as an RL problem, I think using Global Optimization would be a more direct approach. Essentially you want to design a cost function that describes how good a particular seating is and then use it to search the space of possible seatings. For example to solve ...


7

Your understandings are right. deriving the margin to be $\frac{2}{|w|}$ we know that $w \cdot x +b = 1$ If we move from point z in $w \cdot x +b = 1$ to the $w \cdot x +b = 0$ we land in a point $\lambda$. This line that we have passed or this margin between the two lines $w \cdot x +b = 1$ and $w \cdot x +b = 0$ is the margin between them which we ...


6

Although your question is not very specific so I'll try to give you some generic solutions. There are couple of things you can do here: Check sparseMatrix from Matrix package as mentioned by @Sidhha Try running your model in parallel using packages like snowfall, Parallel. Check this list of packages on Cran which can help you runnning your model in ...


6

Here is a trivial example, which captures the essence of genetic algorithms more meaningfully than the polynomial you provided. The polynomial you provided is solvable via stochastic gradient descent, which is a simpler minimimization technique. For this reason, I am instead suggesting this excellent article and example by Will Larson. Quoted from the ...


6

It would be a waste of information; the gradient is available, so use it and save time. There is reason to believe that the local optima are good; see, for example, Choromanska et al. (notes). Over-optimizing for the training set leads to worse generalization, so sometimes we deliberately don't even try by stopping early. Probably the best free lunch in ...


6

One method would be to take many subsets of your dataset, i.e. bootstrapping, build your models, perform cross-validation and calculate the average performance. This is a good explanation of how the amount of data affects the model outcomes: https://stackoverflow.com/questions/25665017/does-the-dataset-size-influence-a-machine-learning-algorithm Play ...


6

This blogpost gives a broad answer to your question. In short, Newton's method is not used to find a root of the loss, but a root of the gradient. If you find a root of the gradient, then you are either in a maximum, a minimum, or a saddle point (the three of them are critical points). When using the cross-entropy loss function in logistic regression, it can ...


5

Learning rate , transformed as "step size" during our iteration process , has been a hot issue for years , and it will go on . There are three options for step size in my concerning : One is related to "time" , and each dimension shall share the same step size . You might have noticed something like $\it\huge\bf\frac{\alpha}{\sqrt{t}}$ while t ...


5

1)AdaGrad penalizes the learning rate too harshly for parameters which are frequently updated and gives more learning rate to sparse parameters,parameters that are not updated as frequently. In several problems often the most critical information is present in the data that is not as frequent but sparse. So if the problem you are working on deals with sparse ...


5

This is the expected behavior. Different learning rates should converge to the same minimum if you are starting at the same location. If you're optimizing a neural network and you want to explore the loss surface, randomize the starting parameters. If you always start your optimization algorithm from the same initial value, you will reach the same local ...


5

Yes the ReLU second order derivative is 0. Technically, neither $\frac{dy}{dx}$ nor $\frac{d^2y}{dx^2}$ are defined at $x=0$, but we ignore that - in practice an exact $x=0$ is rare and not especially meaningful, so this is not a problem. Newton's method does not work on the ReLU transfer function because it has no stationary points. It also doesn't work ...


5

With a higher learning rate, you take bigger steps towards the solution. However, when you are close, you might jump over the solution and then the next step, you jump over it again causing an oscillation around the solution. Now, if you lower the learning rate correctly, you will stop the oscillation and continue towards the solution once again. That is, ...


Only top voted, non community-wiki answers of a minimum length are eligible