Gradient Descent is an algorithm for finding the minimum of a function. It iteratively calculates partial derivatives (gradients) of the function and descends in steps proportional to those partial derivatives. One major application of Gradient Descent is fitting a parameterized model to a set of data: the function to be minimized is an error function for the model.

355 questions
Filter by
Sorted by
Tagged with
24 views

### Multivariable linear gradient descent resulting in inf

I am trying to implement a multivariable gradient descent algorithm, it seems to start working fine, and works on smaller datasets, but applying it to larger datasets the variables overflow and cause ...
38 views

### Gradient descent does not converge in some runs and converges in other runs in the following simple Keras network

When training a simple Keras NN (1 input, 1 level with 1 unit for a regression task) during some runs I get big constant loss that does not change in 80 batches. During other runs it decreases. What ...
22 views

### Why use gradient descent on Deep Nets / RNNs when cost function is not convex?

Why do we use gradient descent on very non-convex loss functions such as in Deep nets / RNNs rather than a heuristic search (genetic algorithms, simulated annealing, etc)?
91 views

### Getting NN weights for every batch / epoch from Keras model

I am trying to get weights for every batch / epoch from Keras model after it is trained. To do so I use callback to make model save weights during training. Yet after model is trained it looks like I ...
29 views

The constrained optimization problem in SVM is given by min 1/2 ||w||^2 s.t y(i)(w^T x(i) + b >= 1 for all i Now converting this to an unconstrained optimization problem gives the lagriangian L as ...
140 views

I was reading about descending gradient. How does the descending gradient know what weights to adjust? Does it adjust to all network weights at the same time? Does each weight have an associated error?...
34 views

### Neural Network Loss Function - Mean Square Error: questions about what 'n' signifies

I'm very new to neural networks and have recently learnt about the loss functions used with neural networks. This question is in regards to the mean square error metric, defined as (from the textbook ...
13 views

22 views

### Issues with self-implemented logistic regression

I am trying to self-implement a logistic regression algorithm to do some self-learning but I am having a bit of trouble with achieving similar accuracy to the logistic regression of sklearn. Here is ...
42 views

According to me: Mini Batch Gradient Descent : 1.It takes a specified batch number say 32. 2.Evaluate loss on 32 examples. 3.Update weights. 4.Repeat until every example is complete. 5.Repeat till a ...
41 views

### With Stochastic Gradient Descent why we dont compute exact derivative of loss function?

In a blog I read this: With Stochastic Gradient Descent we donāt compute the exact derivate of our loss function. Instead, weāre estimating it on a small batch. blog. Now I am confused with the whole ...
25 views

### Why do we use stochastic gradient descent in neural networks and what are the main ideas behind this optimization technique?

I am a student and I am studying machine learning. I am focusing on neural network, and I have seen that for a neural network, to define the optimal weights, we don't use gradient descent, but we use ...
42 views

### Why do we only care about convex functions when doing Gradient Descent/SGD?

I mean I know why we specifically care about convex functions: it's because their local minimum are also global, and so you just have to "follow a path which goes down" to find the minima of ...
50 views

I am trying to implement a gradient descent algorithm for a simple linear function: y(x) = x Where initial hypothesis function is: ...
50 views

### When does it make sense to choose gradient descent for SVM over liblinear?

I understand using gradient descent methods with SVM is intractable if you've used the kernel trick. In that case, best to use libsvm as your solver. But in the case that you are not using a kernel ...
26 views

### implementing forward and backward of a Linear model

I'm implementing the code of this abstraction. The forward is easy and looks like that: I don't understand the backward path and how it fit's the abstraction in the first image: Why is db defined as ...
18 views

### a technique to improve convergence rate

So I saw this technique which is explained as an improvement for gradient descent algorithm, which is based on coordinate transformation: What I don't understand is why in the second slide ...
36 views

### Intuitive explanation for representing gradient in higher dimensions

I do not understand how complex networks with many parameters/dimensions can be represented in a 3D space, and form a standard cost surface just like a simple network with, say, 2 parameters. For ...
10 views

### For calculating gradient penalty, why we need to consider data point that lies on the straight lines between actual and generator data pairs?

I am trying to understand the gradient penalty which was introduced in the following famous paper: Improved Training of Wasserstein GANs Introduced in section 4, equation 3 For calculating gradient ...
6 views

### Why does the error function become constant while implementing stochastic gradient descent using 2D inputs?

According to Q8,9 of HW5, Caltech's Learning from data course, we have to generate 100 test points of the form (x1,x2) and get their outputs 1/0 depending on which side of a random line they lie on. ...
51 views

As I was reading and going through the second course of Andrew Ng's deep learning course, I came across a sentence that said, With a well-turned mini-batch size, usually it outperforms either ...
71 views

### In a neural network, is it possible to gradient descent with more than one input?

I went through a few tutorials, examples recently, and all (not sure if just for demonstration purposes) done gradient descent for one input. To get a deep understanding of backpropagation, I wrote a ...
87 views

### How do local minima occur in the equation of loss function?

In gradient descent, I know that local minima occur when the derivative of a function is zero, but when the loss function is used, the derivative is equal to zero only when the output and the ...
25 views

### What are ways to pick a single training sample to compute gradient in SGD?

instead of computing the full gradient as in GD, SGD computes estimate of the gradient by either computing gradient of a mini-batch or computing gradient for a single training sample, picked randomly ...
9 views

### How to use Meta-SGD with U-Net

I'm guessing how to use Meta-SGD with U-Net network. I've been searching for an example, but I haven't found anything yet. I'm reading the book Hands On Meta Learning with Python, which has the ...
43 views

### Mathematics: Can the result of a derivative for the Gradient Descent consist of only one value?

I have a problem of a task using the formula of the Gradient Descent: Perform two steps of the gradient descent towards a local minimum for the function given below, using a step size of 0.1 and an ...
68 views

### Do batch GD and stochastic GD give the same results?

If a neural network is trained on a dataset of M samples for N epochs, do batch GD and SGD give the same result? Is SGD is faster because utilize the hardware better? I am asking because I figured out ...
79 views

### Learning parameters when loss is a piecewise function

I have a network to generate a single number $T$. I know in advance: a property of the loss function is that, when $T \in [a_1, a_2]$, the loss has the same value $L_1$; when $T \in [a_2, a_3]$, the ...
183 views

The update equation of AdaGrad is as follows: I understand that sparse features have small updates and this is a problem. I understand that the idea of AdaGrad is to make the update speed (learning ...
83 views

### Is it valid to use numpy.gradient to find slope of line as well as slope of curve at any point?

what is the difference between slope of the line and slope of the curve? Is it valid to use numpy.gradient to find the slope of the line and slope of the curve at ...
51 views

### How to find slope of curve at certain points

how to find slope at certain points circled in blue in below curve ? Are these below 2 approaches valid ? though they give different results . How to automatically find the points where the slope ...