Questions tagged [gradient-descent]

Gradient Descent is an algorithm for finding the minimum of a function. It iteratively calculates partial derivatives (gradients) of the function and descends in steps proportional to those partial derivatives. One major application of Gradient Descent is fitting a parameterized model to a set of data: the function to be minimized is an error function for the model.

Filter by
Sorted by
Tagged with
0 votes
0 answers
15 views

Proof that averaging weights is equal to averaging gradients (FedSGD vs FedAvg)

The first paper of Federated Learning "Communication-Efficient Learning of Deep Networks from Decentralized Data" presents FedSGD and FedAvg. In Federated Learning the learning task is ...
3 votes
1 answer
126 views

SGD performing better than Adam in Random minority oversampling, I don't know what is the reason. Help

So my dataset image before and after balancing looks like this: But when I train with Adam(0.0001) and SGD(0.0001), the results are very different. Why? What is going on under the hood? This is ...
0 votes
1 answer
27 views

Gradient descent vs stochastic gradient descent vs mini-batch gradient descent with respect to working step/example

I am trying to understand the working of gradient descent, stochastic gradient descent and mini-batch gradient descent. In case of gradient descent, gradient is computed on the entire dataset at each ...
  • 187
0 votes
1 answer
28 views

Is gradient descent useful to get the least mean squared error in linear regression?

I am new to machine learning. I have read about the linear regression where-in the ideal model is a line which has the least mean squared error. In multi-variable linear regression we would have a ...
  • 187
0 votes
0 answers
12 views

Gradient descent/Adam converging to suboptimal solutions

I am using neural nets to find the minimum of a complex function to which I compute the mean (crit in my code). Here is my net : ...
  • 1
0 votes
1 answer
19 views

why we have problem with gradients when feature values are of different range?

A blog below mentioned. " Because different features do not have similar ranges of values, gradients may take a long time, oscillate back and forth, and take a long time before they can finally ...
0 votes
0 answers
20 views

calculating derivative of bias in backpropagation

Looking at the algorithm in wikipedia, we can implement backpropagation by calculating: $$\delta^{L}=\left(f^{L}\right)'\cdot\nabla_{a^{L}}C$$ (where I treat $\left(f^{L}\right)'$ as an $n\times n$ ...
1 vote
2 answers
24 views

In Gradient descent, Why the gradient of cost function do not have to be normalized into unit vector

From my background, I understand that the purpose of having a learning rate (α) is to normalize the magnitude of gradient (▽J), so the step size can properly converge the local minima Since α is ...
  • 11
0 votes
0 answers
23 views

Why does the sigmoid function being not centered on zero means that the gradient will always be the same sign with positive inputs?

In my course I have this affirmation but I don't see why. It is referring to neural networks' gradient descent.
0 votes
1 answer
47 views

Understanding Conjugate Gradient Optimization methods

As a beginner in ML, I find it hard to understand how Conjugate Gradient Optimization methods work. The sources I've looked up online have a very complicated explanation. Can someone explain in a ...
  • 263
0 votes
0 answers
6 views

How to calculate the expression for the gradient of softmax + cross entropy with respect to weights?

I'm learning cs231n on my own. The Softmax classifier has the following loss function: to make this clear: $L_i$ is the loss for a particular training input $f_j$ is the $j$th element of the vector, ...
  • 115
0 votes
1 answer
39 views

Why would we add regularization loss to the gradient itself in an SVM?

I'm doing CS 231n on my own. I'm looking at this solution to a question that implements a SVM. Relevant code: ...
  • 115
0 votes
1 answer
43 views

What's the logic behind such gradient descent

The gradient descent is motivated from the leetcode question of minimal distance: https://leetcode.com/problems/best-position-for-a-service-centre/ $$\arg\min\limits_{x_c,y_c}\sum\limits_i\sqrt{(x_i-...
1 vote
1 answer
43 views

How does gradient descent avoid local minimums?

In Neural Networks and Deep Learning, the gradient descent algorithm is described as going on the opposite direction of the gradient. Link to place in book. What prevents this strategy from landing in ...
  • 115
0 votes
0 answers
28 views

Understanding Gradient Descent

In Gradient Descent, we start with a random value of the co-efficient & improve upon it with each iteration. Do we move on to the next co-efficient after finding the first one, or do we work on ...
  • 263
0 votes
0 answers
29 views

calculating gradient descent

when using mini batch gradient descent , we perform backpropagation after each batch , ie we calculate the gradient after each batch , we also capture y-hat after each sample in the batch and finally ...
  • 23
0 votes
0 answers
13 views

Is it beneficial to use a batch size > 1 even when all computing power can be used?

In regards to training a neural network, it is often said that increasing the batch size decreases the network's ability to generalize, as alluded to here. This is due to the fact that training on ...
  • 101
1 vote
1 answer
103 views

how to calculate loss function?

i hope you are doing well , i want to ask a question regarding loss function in a neural network i know that the loss function is calculated for each data point in the training set , and then the ...
  • 23
1 vote
1 answer
122 views

Understanding SGD for Binary Cross-Entropy loss

I'm trying to describe mathematically how stochastic gradient descent could be used to minimize the binary cross entropy loss. The typical description of SGD is that I can find online is: $\theta = \...
  • 13
1 vote
2 answers
127 views

'Solvers' in Machine Learning

What role do 'Solvers' play in optimization problems? Surprisingly, I could not find any definition for 'Solvers' online. All the sources I've referred to just explain the types of solvers & the ...
  • 263
-4 votes
1 answer
107 views

Verifying my understanding of MLE & Gradient Descent in Logistic Regression

Here is my understanding of the relation between MLE & Gradient Descent in Logistic Regression. Please correct me if I'm wrong: 1) MLE estimates optimal parameters by taking the partial derivative ...
  • 263
0 votes
0 answers
11 views

How do you find the eigenvalues of the matrix for the following momentum gradient descent?

The following question is based purely on the material available on MIT's open courseware youtube channel. (https://www.youtube.com/watch?v=wrEcHhoJxjM). In it, Professor Gilbert Strang explains the ...
0 votes
0 answers
22 views

ResNet: Derive the gradient matrices w.r.t. W1 and W2 and backprop equation in a Residual Network

How would I go about step by step deriving stochastic gradient matrices w.r.t. W1 and W2 and backpropagation equation in a residual block that is a part of a larger ResNet network with forward ...
  • 1
0 votes
0 answers
29 views

From what function do come the gradients that I use to adjust weights?

I have a question about the loss function and the gradient. So I'm following the fastai (https://github.com/fastai/fastbook) course and at the end of 4th chapter, I got myself wondering. From what ...
2 votes
2 answers
231 views

Difference between OLS and Gradient Descent in Linear Regression

I understand what Ordinary Least Squares and Gradient Descent do but I am just confused about the difference between them. The only difference I can think of are- Gradient Descent is iterative while ...
  • 121
1 vote
0 answers
23 views

Need help to understand the formula of gradient descent with multiple features

I am trying to implement gradient descent with multiple features after listening to Andrew Ng's Coursera lecture. gradient descent for multiple features So for example when calculating for theta 1, ...
  • 11
0 votes
0 answers
10 views

What is "Gradient × Hidden States" explainability method? Is there any documentation about it?

I am doing a literature review on post-hoc explainability methods based on gradient. I stumbled upon one I didn't heard of to extract highlights from a trained model in this post-hoc fashion: We ...
1 vote
0 answers
7 views

How to interpret integrated gradients in an NLP toxic text classification use-case?

I am trying to understand how integrated gradients work in the NLP case. Let $F: \mathbb{R}^{n} \rightarrow[0,1]$ a function representing a neural network, $x \in \mathbb{R}^{n}$ an input and $x' \in ...
1 vote
1 answer
23 views

Understanding Learning Rate in depth

I am trying to understand why the learning rate does not work universally. I have two different data sets and have tested out three learning rates 0.001 ,0.01 and 0.1 . For the first data set, I was ...
  • 11
0 votes
0 answers
15 views

Is there a difference between AutoGrad and explicit derivatives (gradient)?

Will there be some differences between applying AutoGrad on the loss function (using a python library) and applying explicit gradient (the gradient from the paper or the update rule)? For example: ...
  • 1
0 votes
1 answer
93 views

Why does neural network need loss as scalar?

I have a loss function that's a weighted cross entropy loss for binary classification ...
0 votes
0 answers
11 views

Is every timestep when training an LSTM part of the error, or just the end of the sequence?

I'm attempting to write my own LSTM network in C++ for fun. I've already got a regressive and classification network working with regular perceptrons and it works well. What I do currently is divide ...
1 vote
0 answers
25 views

Why it's better to use double floating point precision instead of single point precision for gradient check

I came across the following," A common pitfall is using single precision floating point to compute gradient check. It is often that case that you might get high relative errors (as high as 1e-2) ...
  • 187
0 votes
0 answers
68 views

Training error vs testing error for batch gradient descent on Pyrhon, can't understand what I'm supposed to do

I have this dataset containing training data and testing data , and I have to plot training and testing errors for the gradient descent algorithm with square and logistic losses. I'm a beginner, I'm a ...
0 votes
0 answers
19 views

Gradient descent to minimize the sum of arc length distances on a unit sphere

I have to write a batch gradient descent algorithm to find the point on the unit sphere (in any number of dimensions) which minimizes the sum of arc length distances to each of a set of given points ...
0 votes
0 answers
26 views

How does MultiVariate Linear Regression actually calculate each coefficient?

W​hile calculating the coefficients in gradient descent for multivariate linear regression, the loss accounted in all the coefficients(parameters) updating-equations is the same, loss=predicted-y x_1=...
0 votes
1 answer
23 views

Loss-value of normal equation vs gradient descent

My question is if gradient descent can give a better aproximation than normal equation in Python? for the Loss function, I wrote ...
0 votes
1 answer
24 views

Vanishing gradients: examine output gradients

For a feedforward network or RNN, in theory we should examine the output gradients with respect to the weights over time to check whether it vanishes to zero. In my code below I am not sure whether it ...
  • 113
0 votes
0 answers
98 views

Adding a group specific penalty to binary cross-entropy

I want to implement a custom Keras loss function that consists of plain binary cross-entropy plus a penalty that increases the loss for false negatives from one class (each observation can belong to ...
  • 1
1 vote
2 answers
284 views

MLE & Gradient Descent in Logistic Regression

In Logistic Regression, MLE is used to develop a mathematical function to estimate the model parameters, optimization techniques like Gradient Descent are used to solve this function. Can somebody ...
  • 263
0 votes
0 answers
25 views

Question from a paper: I do not understand why it is stated that SGD employs the bootstrapping to calculate gradient?

In this paper, they state that: As SGD employs the bootstrapping (i.e., random sampling with replacement) [67] for gradient calculation, we can obtain the unbiased estimation of standard gradients ...
  • 121
1 vote
2 answers
683 views

Relation between MLE (Maximum Likelihood Estimation) & Gradient Descent

What are the similarities & dissimilarities between MLE (used to find the best parameters in logistic regression) & Gradient Descent?
  • 263
0 votes
0 answers
7 views

How to overcome extremely variable model performance?

I am training an LSTM and the model performance seems to range from near perfect to dreadful by visual inspection of predicted values - it seems by random initialisation simply retraining the model ...
3 votes
2 answers
249 views

Gradient descent implementation of logistic regression

Objective Seeking for help, advise why the gradient descent implementation does not work below. Background Working on the task below to implement the logistic regression. Gradient descent Derived the ...
  • 513
1 vote
0 answers
13 views

How do I find the gradients of an X -> RELU -> RELU -> Softmax network?

I am trying to use the MNIST digits dataset with logistic regression. I am only using numpy as I would like to implement a simple project myself before using something like pytorch. My network ...
  • 111
1 vote
0 answers
59 views

TypeError: only size-1 arrays can be converted to Python scalars

I am doing multiple linear regression using gradient descent and am getting this type-error ...
0 votes
0 answers
30 views

What stochastic gradient descent is and isn't

I've read what stochastic gradient descent is in multiple different places but everyone describes it in a slightly different way which makes me questions what is really is. Is my definition below ...
  • 101
1 vote
1 answer
57 views

Why does stochastic gradient descent lead us to a minimum at all?

Why do we think that stochastic gradient descent is going to find a minimum at all? I mean on each iteration SGD moves in the direction that reduces only current batch's error (SGD doesn't care about ...
  • 121
2 votes
1 answer
257 views

Does gradient descent always find global minimum for specific regression type?

From my understanding, linear regression is used for predicting an output based on an input using a linear equation that is optimally fitted to some input data. We choose the best fitted linear ...
  • 23
0 votes
2 answers
140 views

How does Gradient Descent work? [duplicate]

I know the calculus and the famous hill and valley analogy (so to say) of gradient descent. However, I find the update rule of the weights and biases quite terrible. Let's say we have a couple of ...

1
2 3 4 5
9