Questions tagged [gradient-descent]

Gradient Descent is an algorithm for finding the minimum of a function. It iteratively calculates partial derivatives (gradients) of the function and descends in steps proportional to those partial derivatives. One major application of Gradient Descent is fitting a parameterized model to a set of data: the function to be minimized is an error function for the model.

Filter by
Sorted by
Tagged with
-1
votes
1answer
40 views

How and why does SGD with Momentum really work? [closed]

Alright, now I am asking this. I have read dozens of articles on SGD with momentum, watched dozens of videos on it. None explained, how and why does it work, properly. I have just created an account ...
1
vote
1answer
20 views

Issues with self-implemented logistic regression

I am trying to self-implement a logistic regression algorithm to do some self-learning but I am having a bit of trouble with achieving similar accuracy to the logistic regression of sklearn. Here is ...
0
votes
2answers
30 views

Why Mini batch gradient descent is faster than gradient descent?

According to me: Mini Batch Gradient Descent : 1.It takes a specified batch number say 32. 2.Evaluate loss on 32 examples. 3.Update weights. 4.Repeat until every example is complete. 5.Repeat till a ...
-1
votes
0answers
25 views

With Stochastic Gradient Descent why we dont compute exact derivative of loss function? [duplicate]

In a blog I read this: With Stochastic Gradient Descent we don’t compute the exact derivate of our loss function. Instead, we’re estimating it on a small batch. blog. Now I am confused with the whole ...
0
votes
1answer
39 views

With Stochastic Gradient Descent why we dont compute exact derivative of loss function?

In a blog I read this: With Stochastic Gradient Descent we don’t compute the exact derivate of our loss function. Instead, we’re estimating it on a small batch. blog. Now I am confused with the whole ...
0
votes
0answers
20 views

Why do we use stochastic gradient descent in neural networks and what are the main ideas behind this optimization technique?

I am a student and I am studying machine learning. I am focusing on neural network, and I have seen that for a neural network, to define the optimal weights, we don't use gradient descent, but we use ...
0
votes
1answer
23 views

Why do we only care about convex functions when doing Gradient Descent / SGD?

I mean, i know why we specifically care about convex functions : it's because their local minimum are also global, and so you just have to "follow a path which goes down" to find the minima ...
0
votes
1answer
39 views

Linear function gradient descent

I am trying to implement a gradient descent algorithm for a simple linear function: y(x) = x Where initial hypothesis function is: ...
3
votes
1answer
43 views

When does it make sense to choose gradient descent for SVM over liblinear?

I understand using gradient descent methods with SVM is intractable if you've used the kernel trick. In that case, best to use libsvm as your solver. But in the case that you are not using a kernel ...
0
votes
1answer
21 views

implementing forward and backward of a Linear model

I'm implementing the code of this abstraction. The forward is easy and looks like that: I don't understand the backward path and how it fit's the abstraction in the first image: Why is db defined as ...
0
votes
0answers
17 views

a technique to improve convergence rate

So I saw this technique which is explained as an improvement for gradient descent algorithm, which is based on coordinate transformation: What I don't understand is why in the second slide ...
0
votes
2answers
29 views

Intuitive explanation for representing gradient in higher dimensions

I do not understand how complex networks with many parameters/dimensions can be represented in a 3D space, and form a standard cost surface just like a simple network with, say, 2 parameters. For ...
0
votes
0answers
10 views

For calculating gradient penalty, why we need to consider data point that lies on the straight lines between actual and generator data pairs?

I am trying to understand the gradient penalty which was introduced in the following famous paper: Improved Training of Wasserstein GANs Introduced in section 4, equation 3 For calculating gradient ...
0
votes
0answers
6 views

Why does the error function become constant while implementing stochastic gradient descent using 2D inputs?

According to Q8,9 of HW5, Caltech's Learning from data course, we have to generate 100 test points of the form (x1,x2) and get their outputs 1/0 depending on which side of a random line they lie on. ...
1
vote
1answer
46 views

Can mini-batch gradient descent outperform batch gradient descent? [duplicate]

As I was reading and going through the second course of Andrew Ng's deep learning course, I came across a sentence that said, With a well-turned mini-batch size, usually it outperforms either ...
0
votes
2answers
59 views

In a neural network, is it possible to gradient descent with more than one input?

I went through a few tutorials, examples recently, and all (not sure if just for demonstration purposes) done gradient descent for one input. To get a deep understanding of backpropagation, I wrote a ...
2
votes
2answers
66 views

How do local minima occur in the equation of loss function?

In gradient descent, I know that local minima occur when the derivative of a function is zero, but when the loss function is used, the derivative is equal to zero only when the output and the ...
0
votes
0answers
20 views

What are ways to pick a single training sample to compute gradient in SGD?

instead of computing the full gradient as in GD, SGD computes estimate of the gradient by either computing gradient of a mini-batch or computing gradient for a single training sample, picked randomly ...
0
votes
0answers
8 views

How to use Meta-SGD with U-Net

I'm guessing how to use Meta-SGD with U-Net network. I've been searching for an example, but I haven't found anything yet. I'm reading the book Hands On Meta Learning with Python, which has the ...
1
vote
1answer
43 views

Mathematics: Can the result of a derivative for the Gradient Descent consist of only one value?

I have a problem of a task using the formula of the Gradient Descent: Perform two steps of the gradient descent towards a local minimum for the function given below, using a step size of 0.1 and an ...
1
vote
1answer
52 views

Do batch GD and stochastic GD give the same results?

If a neural network is trained on a dataset of M samples for N epochs, do batch GD and SGD give the same result? Is SGD is faster because utilize the hardware better? I am asking because I figured out ...
4
votes
1answer
72 views

Learning parameters when loss is a piecewise function

I have a network to generate a single number $T$. I know in advance: a property of the loss function is that, when $T \in [a_1, a_2]$, the loss has the same value $L_1$; when $T \in [a_2, a_3]$, the ...
1
vote
0answers
36 views

Why L2 norm in AdaGrad update equation not L1?

The update equation of AdaGrad is as follows: I understand that sparse features have small updates and this is a problem. I understand that the idea of AdaGrad is to make the update speed (learning ...
1
vote
1answer
30 views

Is it valid to use numpy.gradient to find slope of line as well as slope of curve at any point?

what is the difference between slope of the line and slope of the curve? Is it valid to use numpy.gradient to find the slope of the line and slope of the curve at ...
1
vote
1answer
24 views

How to find slope of curve at certain points

how to find slope at certain points circled in blue in below curve ? Are these below 2 approaches valid ? though they give different results . How to automatically find the points where the slope ...
0
votes
1answer
251 views

No gradients provided for any variable: ['Variable:0', 'Variable:0']

I am using Python 3.6 and Tensorflow 2.0 for the following code for linear regression: ...
3
votes
1answer
39 views

Will stochastic gradient descent converge for multivariate linear regression

I am trying to figure out if stochastic gradient descent for a multivariate linear regression will converge (assuming there is no mini-batching, i.e., the batch size is 1). My guess is yes, based on ...
3
votes
1answer
47 views

Why Gaussian mixture model uses Expectation maximization instead of Gradient descent?

Why Gaussian mixture model uses Expectation maximization instead of Gradient descent? What other models uses Expectation maximization to find best optimal parameters instead of using gradient descent?
0
votes
0answers
6 views

what are Latent model and Latent Varibles?

what are latent models and why they use Expectation maximization instead of gradient descent to optmize the parameters ? How to interpret and understand whether a hidden varible is present in model or ...
2
votes
1answer
64 views

How similar is Adam optimization and Gradient clipping?

According to the Adam optimization update rule: $$m \leftarrow \beta_1 m + (1 - \beta_1)\nabla J(\theta)$$ $$v \leftarrow \beta_2 v + (1 - \beta_2)(\nabla J(\theta) \odot \nabla J(\theta))$$ $$\theta \...
3
votes
2answers
139 views

Difference between RMSProp and Momentum?

Can someone please tell me the clear difference between the approaches of RMSProp and Gradient Descent with Momentum ? Both try to achieve the same effect . One of the blogs that I read states the ...
2
votes
1answer
25 views

XGBoost is it possible to prevent a feature from being used twice in the same tree?

I'm using XGBoost and all its doing is using the feature in the first column of my data. My feature importance chart correlates perfectly to the position of the feature in my xtrain. If I shuffle the ...
0
votes
0answers
40 views

clarification on back-propagation calculations for a fully connected neural network

I am currently taking Andrew Ng's Deep Learning Course on coursera and I couldn't get my head around how actually back-propagation in calculated. Let's say my fully connected neural network looks like ...
0
votes
0answers
18 views

Vectorised Neural Network Backprapogation

I've started the CS231n course and have been going through the proofs and derivations for backpropagation. These two papers by the lecturers are what I've been reading: Paper 1: - http://cs231n....
0
votes
1answer
23 views

Neural Network not learning when more than 1 training data is given

I am very new to neural networks and data science in general and wanted to try getting my hand in making a simple neural network in python. I tried to make a neural network from scratch hoping to ...
0
votes
0answers
14 views

When scaled conjugate gradient backpropagation is suitable for backpropagation?

I usually use gradient descent with Adam optimizer to perform backpropagation in deep learning methods. I knew it is a very efficient method. The question is in which situations we can use "scaled ...
0
votes
1answer
32 views

Why does Siamese neural networks use tied weights and how do they work?

Reading this paper on one-shot learning "Siamese Neural Networks for One-shot Image Recognition" I was introduced to the idea of Siamese Neural Networks. What I did not fully grasp was what they ...
1
vote
1answer
20 views

Has Gradient decent any effects on the actual predictions or designed for practical convenience?

If we consider Batch Gradient, Stochastic Gradient, Mini-batch Gradient, will they affect on the actual predictions? as we know they always attempt to reach a local minimum.
1
vote
0answers
25 views

Derivative of Loss wrt bias term

I read this and have an ambiguity. I try to understand well how to calculate the derivative of Loss w.r.t to bias. In this question, we have this definition: ...
0
votes
1answer
32 views

Neural Net Backprop Weight updating Pseudo code help please

Here is my code for Backpropagation weight updating. It's a simple network with 1 hidden layer and 1 output neuron. The activation function of both hidden and output layer uses tanh. I propagate the ...
1
vote
0answers
17 views

Tuning parameters for gradient boosting/xgboost

In practice, which parameter do you typically tune first? Do you tune the learning rate (or step size) first? and then tune the total number of iterations? And how do you go about tuning these ...
0
votes
1answer
19 views

how does gradient descent update weigths in neural network

Im currently trying to learn about back propagation, and it's going forward, but theres one thing that keeps me scratching my head, and doesnt really seems to be answered in any of the videos or ...
0
votes
0answers
31 views

How to compute error (instead of gradient of error) for each node in backpropagation?

I was going through the relevant chain rule mathematics and I have successfully implemented backpropagation from scratch for MNIST (once, I even tried doing this for a small sample data I created by ...
0
votes
0answers
14 views

Averaging biased gradient information?

Consider the following scenario: I want to estimate the gradient at a point $P$ in $\mathbb{R}^2$, and I have access to two pieces of directional information, the vectors $u$ and $v$. When we can ...
0
votes
1answer
32 views

Gradient of a function in Python

I've defined a function in this way: def qfun(par): return(par[0]+atan(par[3])*par[1]+atan(par[4])*par[2]) How can I obtain the gradient of this function ...
1
vote
1answer
29 views

Why is my loss increasing in gradient descent?

When the learning rate is 0.01 the loss seems to be decreasing whereas when I increase the learning rate even slightly, the loss increases. Why does this happen? Are the gradients calculated wrong? ...
0
votes
0answers
33 views

'loss: nan' during training of Neural Network in python

I am training a neural network in python. But the accuracy is very low. ...
1
vote
1answer
34 views

What is the gradient descent rule using binary cross entropy (BCE) with tanh?

Similar to this post, I need the gradient descent step of tanh but now with binary cross entropy (BCE). So we have $$ \Delta \omega = -\eta \frac{\delta E}{\delta \omega} $$ Now we have BCE: $$ ...
0
votes
0answers
14 views

Convergence check of constrained biconvex optimization problem

I participate in the development of a matrix factorization algorithm and I have some convergence issues. Here is the kind of minimization problem I am facing : $L(D,A) = ||X-DA||^2 + \sum_{(i,j) \in ...
0
votes
1answer
90 views

Mini Batch Gradient Descent shuffling

My data set is of shape (60,784,1000) with mini batches for input and (60,10,1000) for labels, should I shuffle only the 60 mini batches or the training examples themselves?

1
2 3 4 5
7