Skip to main content
OverflowAI is here! AI power for your Stack Overflow for Teams knowledge community. Learn more

Questions tagged [gradient-descent]

Gradient Descent is an algorithm for finding the minimum of a function. It iteratively calculates partial derivatives (gradients) of the function and descends in steps proportional to those partial derivatives. One major application of Gradient Descent is fitting a parameterized model to a set of data: the function to be minimized is an error function for the model.

Filter by
Sorted by
Tagged with
0 votes
1 answer
23 views

Using a very very small learning rate to not diverge?

i just started with machine learning and today i tried implementing the gradient descent algorithm for linear regression. If i use a bigger value for alpha(the learning rate) the absolute value of w ...
Foch29's user avatar
  • 1
0 votes
1 answer
28 views

Backtracking line search in gradient descent for a function with inputs of multiple dimensions

I am trying to use gradient descent to minimize a function that takes in multiple vectors, so something like $\min f(x_1, x_2,.., x_N)$ where each $x_i \in \mathbb{R}^3$. and the output is a scalar. I'...
confused's user avatar
1 vote
1 answer
41 views

How to comment on goodness of loss functions?

I have two loss functions $\mathcal{L}_1$ and $\mathcal{L}_2$ to train my model. The model is predominantly a classification model. Both $\mathcal{L}_1$ and $\mathcal{L}_2$ takes are two variants of ...
Aleph's user avatar
  • 123
1 vote
0 answers
49 views

Negative MNIST - Trying to understand what's going on here

I'm new to machine learning and pytorch and any insight would be greatly appreciated. I've been playing around with the MNIST number recognition dataset and thought it would be interesting to train ...
Sloth Man's user avatar
  • 111
0 votes
0 answers
23 views

Clarifying the arguments of "Understanding the difficulty of training deep feedforward neural networks"

I made the decision to try to push through the paper "Understanding the difficulty of training deep feedforward neural networks". (The paper is given as a reference in "Hands-On Machine ...
Chris's user avatar
  • 101
1 vote
1 answer
31 views

In a Computational Graph, how to calculate the total upstream gradient of a node with multiple upstreams?

Given a Computation Graph with a node (like the one below), I understand that I can use the upstream gradient dL/dz to calculate all of my downstream gradients. But what if there are multiple ...
Ibrahim's user avatar
  • 111
0 votes
1 answer
23 views

Does learning rate depend on input and output range?

I watched hours of videos on gradient descent and still feel pretty confused. Let's say I have a "model": y = x * w I use 2 as my target ...
Eugene's user avatar
  • 103
0 votes
0 answers
21 views

Mean Square Error not decreasing during gradient descent

I am currently writing a program which is supposed to implement gradient descent to train a prediction model. I am encountering an error whereby my MSE continuously increases, and the network never ...
Zaheer 's user avatar
1 vote
0 answers
30 views

Standardizing my target versus not-standardizing

I've heard from multiple sources that it depends on whether I should standardize or not. Most of the time, people would say it doesn't make sense to do so, some would say it's better if I standardize ...
Justin Jonany's user avatar
0 votes
0 answers
19 views

Minimize MAE loss for a target that is sum of two other targets

Working on a regression modelling task where my dataset have some feature columns, two more columns A, B and a target column T. The goal is to predict T, and minimize MAE, that is ...
Ngo Cuong's user avatar
1 vote
2 answers
180 views

Gradient Descent: Is the magnitude in Gradient Vectors arbitrary?

I am only just getting familiar with gradient descent through learning logistic regression. I understand the directional component in the gradient vectors is correct information derived from the slope ...
MrHunda's user avatar
  • 11
0 votes
0 answers
17 views

Why does weight decay produce regularisation in Deep Neural Network?

Weight decay penalizes the model to have smaller weights but how does this help in regularisation? Any intuition on smaller weights => simpler model?
Sushil Khadka's user avatar
1 vote
0 answers
51 views

Confusion over taking gradients in Variational Autoencoder (VAE)

I am confused as to when to hold certain parameters constant in a VAE. I will explain with a concrete example. We can write $\operatorname{ELBO}(\phi, \theta) = \mathbb{E}_{q_{\phi}(z)}\left[\log \...
Joel's user avatar
  • 11
0 votes
0 answers
43 views

Gradients of lower layers of a CNN when gradient of an upper layer is 0?

Say we have a convolutional neural network with an input layer, 3 convolutional layers and an output layer. Say the gradients with respect to the weights and biases of the third convolutional layer ...
VJ123's user avatar
  • 147
2 votes
1 answer
370 views

Gradients of lower layers of NN when gradient of an upper layer is 0?

Say we have a neural network with an input layer, a hidden layer and an output layer. Say the gradients with respect to the weights and biases of the output layer are all 0. Then, by backpropagation ...
VJ123's user avatar
  • 147
0 votes
0 answers
22 views

How to calculate the gradient magnitude , orientation and threshold the image intensity values = [10 20 5],[10 20 5] kx=[1 -1] ky=[1 ], [-1]

How to calculate the gradient magnitude , orientation and threshold the image intensity values = [10 20 5],[10 20 5] kx=[1 -1] ky=[1 ], [-1] image intensity values = [10 20 5] [10 20 ...
Inzamam ul haq's user avatar
0 votes
0 answers
47 views

Backpropagation and Gradient Descent: Questions on math behind it

I watched this video which goes over backpropagation calculus and read the Wikipedia page on it. This is my understanding of the equations for the algorithm. I have questions regarding the equations ...
notaorb's user avatar
  • 101
1 vote
1 answer
75 views

Why Cost function is differentiable?

I've a very basic question about cost functions. I'm studying gradient descent and there we're using partial differentiation of features "Theta". But isn't the cost function an absolute ...
MLENGG's user avatar
  • 11
0 votes
2 answers
69 views

Does LinearRegression uses Gradient Descent for finding slope and y-intercept of the best fit line?

I know that Gradient Descent is an optimization algorithm used for optimizing the cost of the loss function. Does Linear Regression model of the sklearn package use ...
mainak mukherjee's user avatar
0 votes
0 answers
161 views

TensorFlow Gradient Tape - LookupError (Tandem Neural Network project)

I am trying to train a network by using a custom lost function, then computing the gradient of the loss, and updating the trainable variables in the reverse network (named: reverse_loaded). For ...
user151125's user avatar
0 votes
1 answer
13 views

feature engineering mechanism

why do we need to rescale some feature having large range I know we do it for faster rate of gradient descent ,but still how does rescaling works? and it doesn't break the model and does rescaling ...
rushi jhala's user avatar
1 vote
1 answer
48 views

Doubt in gradient , vanishing gradient problem in Back propagation

As per my knowledge, in back propagation- loss function or gradient is used to update the weights. in back propagation, weights became small w.r.t gradients, this leads to vanishing gradient problem. ...
tovijayak's user avatar
0 votes
0 answers
13 views

Correlation between hinge loss function, Langrage function and ai

The function $f(w,b) = \frac{1}{2} ||w||^2$ is our objective function while our constraints are all the correct classifications of the data points expressed as $g(w,b) = \sum_{i=1}^{l} (y_i (x_i \cdot ...
eisa.exe's user avatar
0 votes
1 answer
52 views

Gradients Exploding with Custom Gradient and 2 layer MLP

I am training a 2 layer MLP on an off-policy learning-to-rank task, where the input is a list of documents against a query with a feature vector for each query-document pair, i.e. if there are M ...
SHASHANK GUPTA's user avatar
0 votes
0 answers
28 views

cost function including constrain on vector norm

I'm playing with collaborative implementation using numpy. As a reminder, we are given a matrix $R$ of user ratings for movies. Let's assume there are 3 users and 4 movies. The data matrix we are ...
ed WSA's user avatar
  • 1
0 votes
0 answers
38 views

Exploring the Concept of Gradient Flow

Understanding the concept of "Gradient Flow" can be quite difficult as there is a lack of widely recognized and clearly defined resources that provide a comprehensive explanation. Although ...
StudentV's user avatar
2 votes
2 answers
4k views

What exactly is Gradient norm?

I found that there is no common resource and well defined definition for "Gradient norm", most search results are based on ML experts providing answers which involves gradient norm or papers ...
StudentV's user avatar
0 votes
1 answer
70 views

How exactly does the mini batch method work?

I mean let's say I have a mini batch, I take an example from it and for it I do the following: I do forward propagation. Using the output after forward propogation - I calculate the gradients of the ...
David's user avatar
  • 11
0 votes
1 answer
144 views

Affine layer - gradient shape

In course cs231n, I need to implement backward pass computation for an affine (linear) layer: ...
Ben Lahav's user avatar
1 vote
2 answers
44 views

Gradient vector starts to increase at some point, gradient descent from scratch

I have a simple linear function y = w0 + w1 * x, where w0 and w1 are weights, And I'm trying to implement a gradient descent for it. I wrote the function and tested in on the data(a dataset of two ...
Clarify's user avatar
  • 13
0 votes
1 answer
26 views

moments of weight vectors in Adam

When performing backpropagation with Adam algorithm, are the moment and the second moment of the weight vectors calculated also for the weights in hidden layers?
Iya Lee's user avatar
  • 152
0 votes
1 answer
33 views

what is static objective function

Paper on Adam mentions Stationary Objective function. I am not able to find its definition on Internet (or may be it's there with some other name and I am not able to figure it out). I will be ...
Saurabh Jain's user avatar
0 votes
1 answer
29 views

AdaGrad denominator exponents comparison

Adagrad adapts the learning rate $ \alpha $ all along the gradient descent process, by dividing each weight on a quantity based on the sum of the previous squared gradient up to time $ t $. Therefore, ...
The Limit Does Not Exist's user avatar
1 vote
0 answers
23 views

How to calculate gradient of MSE in backpropagation? [duplicate]

I want to implement a neural network from scratch to solve linear regression by using backpropagation. I don't understand how to compute the gradient of the MSE cost function with respect to each ...
Iya Lee's user avatar
  • 152
0 votes
1 answer
406 views

Why is cross entropy loss averaged and not used directly as a sum during model training(such as in neural networks)

Why is the cross entropy loss for all training examples(or the training examples in a batch) averaged over size of the training set(or batch size) ? Why is it not just summed and used ?
user17420392's user avatar
1 vote
1 answer
45 views

How train a pre-trained model based on new dataset?

I have trained a deep nn model based on some existing data. In the meantime, I have collected more data and label them so that I can feed it to the model to improve its performance. The questions is, ...
Mahdi Amrollahi's user avatar
0 votes
1 answer
115 views

GAN Generator Backpropagation Gradient Shape Doesn't Match

In the TensorFlow example (https://www.tensorflow.org/tutorials/generative/dcgan#the_discriminator) the discriminator has a single output neuron (assume batch_size=1). Then over in the training loop ...
rkuang25's user avatar
0 votes
1 answer
331 views

Understanding the stochastic average gradient (SAG) algorithm used in sklearn

For pedagogical purposes I've been trying to create my own implementation of the stochastic average gradient (SAG) algorithm in a logistic regression framework. Page 10 of the associated paper ...
hillard28's user avatar
0 votes
1 answer
120 views

What does the ellipse in Gradient descent describes?

I saw multiple articles describes GD or SGD with the following diagram: I didn't saw any explanation about the ellipses. What ...
user3668129's user avatar
1 vote
0 answers
261 views

How Adam optimizer influence the learning rate? [closed]

I read some papers about how ADAM optimizer works, and there are some issues which seems that are confusing: ADAM equations are: ...
user3668129's user avatar
0 votes
1 answer
70 views

What will happen if we apply Gradient Ascent?

I have built a simple neural network on MNIST, but instead of moving toward the opposite direction of gradients, I moved in the same direction of it just by applying( pytorch ): ...
Mahdi Amrollahi's user avatar
2 votes
2 answers
966 views

Does settings $\beta_1 = 0$ or $\beta_2 = 0$ means that ADAM behaves as RMSprop or Momentum?

I read on ADAM optimizer, and I saw multiple quotes which say that ADAM is a combination of Momentum and RMSprop optimizers. So if we: Set $\beta_1 = 0$ does it means that ADAM behaves exactly as ...
user3668129's user avatar
0 votes
1 answer
101 views

Normalising data for simple linear regression

Consider a simple linear regression problem where: X = [1,2,3,4,5,100,200] Y= [2,4,6,8,10,200,400] Clearly, the relationship is of the form $y=2x$; While trying ...
butwhy's user avatar
  • 13
6 votes
4 answers
1k views

How to fit a math formula to data?

I have math formula and some data and I need to fit the data to this model. The math is like. $y(x) = ax^k + b$ and I need to estimate the $a$ and the $b$. I have triet gradient descend to estimate ...
Mahdi Amrollahi's user avatar
0 votes
1 answer
517 views

Why does Adam outperform SGD in logistic regression?

I am training a logistic regression model. In case it matters, the features are 1376-dimensional embeddings output from a neural network. I tried both SGD and Adam with a learning rate of $10^{-3}$ ...
nalzok's user avatar
  • 113
0 votes
1 answer
416 views

Difference between sklearn's LogisticRegression and SGDClassifier?

What is the difference between sklearn's LogisticRegression classifier and its SGDClassifier? I understand that the SGD is an ...
BigBrownBear00's user avatar
0 votes
0 answers
83 views

Why backpropagation is done in every epoch when loss is always scalar?

I understand the backpropagation algorithm that it calculates the derivate of loss with respect to all the parameters in the neural network. My question is this derivate is constant right because the ...
Jeet's user avatar
  • 101
0 votes
0 answers
57 views

Question about input value in Gradient descent

I am currently going through Udacity´s online course "Intro to Deep Learning with PyTorch". In one of the videos covering the Gradient descent algorithm they show the formula for how the ...
Leo's user avatar
  • 1
0 votes
1 answer
251 views

Setting shape of RaggedTensor with Known Shape

I'm working with RaggedTensors to manipulate a dense tensor. Something like this : ...
Liam F-A's user avatar
2 votes
0 answers
83 views

Can I find the input that maximises the output of a Neural Network?

So I trained a 2 layer Neural Network for a regression problem that takes $D$ features $(x_1,...,x_D)$ and outputs a real value $y$. With the model already trained (weights optimised, fixed), can I ...
puradrogasincortar's user avatar

1
2 3 4 5
10