Questions tagged [gradient-descent]

Gradient Descent is an algorithm for finding the minimum of a function. It iteratively calculates partial derivatives (gradients) of the function and descends in steps proportional to those partial derivatives. One major application of Gradient Descent is fitting a parameterized model to a set of data: the function to be minimized is an error function for the model.

Filter by
Sorted by
Tagged with
-1
votes
0answers
18 views

LBFGS Solver query

Lbfgs definition on the internet: Stands for Limited-memory Broyden–Fletcher–Goldfarb–Shanno. It approximates the second derivative matrix updates with gradient evaluations. Can somebody please ...
1
vote
1answer
40 views

Why does stochastic gradient descent lead us to a minimum at all?

Why do we think that stochastic gradient descent is going to find a minimum at all? I mean on each iteration SGD moves in the direction that reduces only current batch's error (SGD doesn't care about ...
2
votes
1answer
38 views

Does gradient descent always find global minimum for specific regression type?

From my understanding, linear regression is used for predicting an output based on an input using a linear equation that is optimally fitted to some input data. We choose the best fitted linear ...
0
votes
0answers
8 views

Formula of momentum gradient descent optimizer

Learning about the optimizers recently, I was confused about the formula for momentum. I mean, I understood the concept but I came across the following 2 formulas while learning. I see that the left ...
0
votes
2answers
66 views

How does Gradient Descent work? [duplicate]

I know the calculus and the famous hill and valley analogy (so to say) of gradient descent. However, I find the update rule of the weights and biases quite terrible. Let's say we have a couple of ...
0
votes
1answer
24 views

Understanding intution behind sigmoid curve in the context of back propagation

I was trying to understand significance of S-shape of sigmoid / logistic function. The slope/derivative of sigmoid approaches zero for very large and very small input values. That is $σ'(z) ≈ 0$ for $...
1
vote
0answers
10 views

RMSprop in weight update - what if vertical slopes small and horizontal slopes large?

I have a question regarding the intuition behind RMSprop, As shown in the lecture video of Deep Learning Specialization by Andrew Ng, RMSprop helps to reduce the oscillation (the values of the ...
1
vote
0answers
17 views

Is saddle point a cause for the vanishing gradient problem

I am a beginner to neural networks and I am writing a report summarising on the causes and solutions to the vanishing gradient problem. From what I have read, the 2 main causes are the repeated ...
0
votes
0answers
7 views

cost function diverging in batch gradient descent

I am trying to implement the gradient descent method in python. I would like the calculation to stop when abs(J-J_new) reaches a certain tolerance level (i.e. it ...
0
votes
0answers
32 views

Gradient descent in linear regression converges but the trend line is incorrect

For the dataset https://physics.info/linear-regression/dash-world.txt, I have been trying to implement linear regression for predicting the men record times as a function of year. I have used gradient ...
0
votes
0answers
14 views

Matrix multiplication

I have downstream gradient for every $sample$ (each row for every $x_i$) $$ \begin{bmatrix} 0.0062123 & -0.00360166 & -0.00479891 \\ -0.01928449 & 0.01240768 & 0.01493274 \\ ...
0
votes
0answers
22 views

How to check for vanishing gradient in CNN?

Is there is a way to check for vanishing gradient for CNNs using Keras? Like, for example, drawing the weight distributions for each layer and seeing if they are vanishing to zero?
0
votes
0answers
20 views

Andrew Ng Deep Learning Gradient Descent of Softmax is just y_hat - y?

At about 8:30 in the video here: https://www.youtube.com/watch?v=ueO_Ph0Pyqk so for the given example with 4 classes and first ground truth y being [0,1,0,0] and y_hat being [0.3,0.2,0.1,0.4] for ...
0
votes
1answer
22 views

Linear regression and gradient descend equations

I'm pretty new to ML and was starting out with linear regression combined with gradient descend. This is the equation I was trying to achieve using javascript- And this is what I came up with in js- <...
1
vote
2answers
34 views

How can the ReLU function lead to convergence?

The gradient descent algorithm is based on the fact that the gradient decreases as we move towards the optimum point. However, in the activations by the ReLU ...
1
vote
1answer
51 views

Vanishing gradient problem even after existence of ReLu function?

Let's say I have a deep neural network with 50 hidden layers and at each neuron of hidden layer the ReLu activation function is used. My question is Is it possible for vanishing gradient problem to ...
0
votes
0answers
23 views

Neural Net gradient descend

I was planning on making my own neural network library in C++ and was going through other's code to make sure I am on right track. Below is a sample code that I am trying to learn from. Everything in ...
1
vote
0answers
10 views

Compare rate of change for multiple object/weights

For a Neural Network, the weight update equation is: However, there are millions of such weights W_i. If I am interested in capturing how much each weight/connection W_i is changing as compared to ...
1
vote
0answers
24 views

Stochastic Gradient Region of Confusion

I have come across the following diagram which explains the behavior of SGD graphically. Based on this graphical representation, the gradient of the individual data tend to fluctuate more when it ...
0
votes
1answer
11 views

Gradient descent different implementation cause error

We know that we can get closer to the local minimum of the function by descending our argument according to that rule $$w1 = w0 − γ∇f$$ For example I have a linear regression model that depends on $b,...
1
vote
1answer
33 views

Gradient descent method

If we suppose that this is formula for gradient descent method: $$x_{n+1}=x_n-\lambda\cdot{{df(x)}\over{dx}},\ n=0,1,2,3,...$$ Since there is no exact value that we subtract instead of derivative, ...
0
votes
1answer
31 views

Why the gradient of a ReLU for X>0 is 1?

Gradient is derivative of several variables. I can't understand why is the gradient of a ReLU for X>0 is 1 ? and 0 for x < 0 ? I tried to search for proof and examples but didn't found any good ...
0
votes
0answers
13 views

Step size finds by quadratic fitting in steepest descent

I have a function $f=(1-x_1)^2 + (x_2-(x_1^2))^2$ and initial point $[0,5]$. I wonder how I will find step size by quadratic fitting using the (e.g. $0.01$) value in Steepest Descent with Matlab. To ...
0
votes
0answers
17 views

Bug in single layer Adaline Neuron implementation

I am trying to implement a single layer Adaline neuron, with the following mathematical foundation: The cost function is defined as: The weight update is defined as: inserting the partial derivative ...
0
votes
0answers
10 views

Method for Finding All Local Extrema Using Gradent Ascent/Descent

I have a very abstract model where a set of coefficients controls animal behavior. This model is so abstract that the actual values of a global extrema are not particularly interesting. However, the ...
1
vote
2answers
21 views

Why does using Gradient descent over Stochatic gradient descent improve performance?

Currently, I'm running two types of logistic regression. logistic regression with SGD logistic regression with GD implemented as follows ...
0
votes
0answers
19 views

Getting gradient for gradCam in pytorch

I am using forward and backward hook in my pytorch densenet121 model. I set requires_grad to False at the time of training. ...
0
votes
0answers
6 views

what would happen in max_pool layer if backprop would add gradient to all inputs for particular neuron but only if it's positive

in max_pool layer ANN performs this operation max([in1, in2, ... inN]), now if gradient that comes back to this layer is ...
0
votes
1answer
21 views

Is the usage of the "momentum" significiantly superior to the conventional weight update

The "momentum" adds a little of the history of the last weight updates to the actual update, with diminishing weight history (older momentum shares get smaller). Is it significiantly ...
0
votes
0answers
15 views

Gradient Descentfor non-linear predictor?

From I what I have read so far, if a loss function $\mathcal{l}(y,\hat{y})$ is convex where $\hat{y}$ is the (to be estimated) decision function, then gradient descent tries to minimizes the loss ...
0
votes
0answers
6 views

Gradient calculation of the pre-trained model

I have pre-trained tensorflow model in a graph format. I have not used tf.gradient on the graph structure. In such cases, is there any way to calculate the gradient of some tensors operation?
1
vote
1answer
19 views

Why do neural networks use cost minimization of loss function and not profit maximization of profit function?

In neural networks, gradient descent is used to find optimum minimum value of cost function. Why this preference instead of finding maximum value of profit function? What are the pros and cons of ...
0
votes
0answers
15 views

Theano error when performing Linear Regression

I'm trying to perform Linear Regression using Theano, but there is something I might be missing or doing wrong because I receive an error message, here you have a reproducible example: ...
2
votes
1answer
50 views

What method/algorithm for constrained multi-target regression

I am working with three dimensional measurement data and want to model them using a multivariate linear regression. I have already implemented a simple gradient descent algorithm to solve the classic ...
1
vote
1answer
26 views

What issue is there, when training this network with gradient descent? [closed]

Suppose we have the following fully connected network made of perceptrons with a sign function as the activation unit, what issue arises, when trying to train this network with gradient descent?
0
votes
1answer
27 views

Vanishing gradient problem

In a neural network, does gradient vanish during a great number epochs as well, rather that only vanishing through different layers?
0
votes
1answer
28 views

Why is backpropagation used for finding the loss gradient?

I am relatively new to the world of machine learning. After getting a general idea of the concept, I tried creating a program for training a deep learning network from scratch. My goal was to use as ...
0
votes
1answer
45 views

How do i get the loss function graph?

I used Mini-batch gradient descent to train the model, but i am unable to get the proper loss graph. The loss graph is always showed as a straight line. I know there is something wrong but would ...
0
votes
1answer
26 views

Gradient descent around optimal loss surface

All the loss surface used in examples have some of bowl shape that decrease drastically far from the optimal and decrease slowly around the optimal flat point. My questions are: Has all the loss ...
0
votes
0answers
12 views

Learning Rule fo bias weights

Consider the network : The learning rule (weight update )used for the hidden-to-output weights is: The learning rule (weight update )used for the input-to-hidden weights is: So what about the bias ...
0
votes
1answer
43 views

How does the equation "dW = - (2 * (X^T ).dot(Y - Y_hat)) / m" comes in Linear Regression (using Matrix + Gradient Descent)?

I was trying to code the Linear Regression in Python using Matrix Multiplication method using Gradient Descent and followed a code where there was no mention what is the loss but just a code as Per ...
0
votes
0answers
15 views

When to use Gradient boosting over stochastic gradient boosting

Gradient boosting works on the Gradient Descent concept and it's one of the ensemble methods. It has a regularization parameter to select subsamples, which is called stochastic gradient boosting. ...
2
votes
1answer
43 views

Adam Optimiser First Step

Plotting the paths on the cost surface from different gradient descent optimisers on a toy example, I found that the Adam algorithm does not initially travel in the direction of steepest gradient (...
1
vote
1answer
50 views

Derivative of a custom loss function with the logistic function

I have costum loss function with $\mu ,p, o, u, v$ as variables and $\sigma$ is the logistic function. I need to derive this loss function. Due to multiple variables in the loss function, I need to ...
1
vote
0answers
13 views

Can we talk about vanishing activations?

When updating the weights of a deep neural network using backpropagation, to update the weights of a given hidden layer, we use both the partial derivatives of the objective function with respect to ...
0
votes
1answer
24 views

Why are mini-batches degrading my conv net MNIST classifier?

I have made a convolutional neural network from scratch in python to classify the MNIST handwritten digits (centralized). It is composed of a single convolutional network with 8 3x3 kernels, a 2x2 ...
0
votes
0answers
19 views

When should I update weights and biases in Neural Network?

So, I am building a Neural Network from scratch for (typically) classifying MNIST digits. Everything is going fine, I can get up to 85% accuracy accross all testing data with stochastic gradient ...
0
votes
0answers
17 views

Unbiased Predictions for all Distinct Training Subsets

Suppose I have a data set $\left(X_i \in \chi, y_i \in \zeta \right)$ where $X_i$ and $y_i$ correspond to instances and labels, and $\chi$ and $\zeta$ correspond to the space where $X_i$ and $y_i$ ...
6
votes
1answer
488 views

Changing the batch size during training

The choice of batch size is in some sense the measure of stochasticity : On one hand, smaller batch sizes make the gradient descent more stochastic, the SGD can deviate significantly from the exact ...

1
2 3 4 5
8