# Questions tagged [gradient-descent]

Gradient Descent is an algorithm for finding the minimum of a function. It iteratively calculates partial derivatives (gradients) of the function and descends in steps proportional to those partial derivatives. One major application of Gradient Descent is fitting a parameterized model to a set of data: the function to be minimized is an error function for the model.

413 questions
Filter by
Sorted by
Tagged with
15 views

### Proof that averaging weights is equal to averaging gradients (FedSGD vs FedAvg)

The first paper of Federated Learning "Communication-Efficient Learning of Deep Networks from Decentralized Data" presents FedSGD and FedAvg. In Federated Learning the learning task is ...
126 views

### SGD performing better than Adam in Random minority oversampling, I don't know what is the reason. Help

So my dataset image before and after balancing looks like this: But when I train with Adam(0.0001) and SGD(0.0001), the results are very different. Why? What is going on under the hood? This is ...
27 views

### Gradient descent vs stochastic gradient descent vs mini-batch gradient descent with respect to working step/example

I am trying to understand the working of gradient descent, stochastic gradient descent and mini-batch gradient descent. In case of gradient descent, gradient is computed on the entire dataset at each ...
• 187
28 views

### Is gradient descent useful to get the least mean squared error in linear regression?

I am new to machine learning. I have read about the linear regression where-in the ideal model is a line which has the least mean squared error. In multi-variable linear regression we would have a ...
• 187
12 views

### Gradient descent/Adam converging to suboptimal solutions

I am using neural nets to find the minimum of a complex function to which I compute the mean (crit in my code). Here is my net : ...
19 views

### why we have problem with gradients when feature values are of different range?

A blog below mentioned. " Because different features do not have similar ranges of values, gradients may take a long time, oscillate back and forth, and take a long time before they can finally ...
• 11
20 views

### calculating derivative of bias in backpropagation

Looking at the algorithm in wikipedia, we can implement backpropagation by calculating: $$\delta^{L}=\left(f^{L}\right)'\cdot\nabla_{a^{L}}C$$ (where I treat $\left(f^{L}\right)'$ as an $n\times n$ ...
• 101
1 vote
24 views

### In Gradient descent, Why the gradient of cost function do not have to be normalized into unit vector

From my background, I understand that the purpose of having a learning rate (α) is to normalize the magnitude of gradient (▽J), so the step size can properly converge the local minima Since α is ...
• 11
23 views

### Why does the sigmoid function being not centered on zero means that the gradient will always be the same sign with positive inputs?

In my course I have this affirmation but I don't see why. It is referring to neural networks' gradient descent.
47 views

### Understanding Conjugate Gradient Optimization methods

As a beginner in ML, I find it hard to understand how Conjugate Gradient Optimization methods work. The sources I've looked up online have a very complicated explanation. Can someone explain in a ...
• 263
6 views

### How to calculate the expression for the gradient of softmax + cross entropy with respect to weights?

I'm learning cs231n on my own. The Softmax classifier has the following loss function: to make this clear: $L_i$ is the loss for a particular training input $f_j$ is the $j$th element of the vector, ...
• 115
39 views

### Why would we add regularization loss to the gradient itself in an SVM?

I'm doing CS 231n on my own. I'm looking at this solution to a question that implements a SVM. Relevant code: ...
• 115
43 views

### What's the logic behind such gradient descent

The gradient descent is motivated from the leetcode question of minimal distance: https://leetcode.com/problems/best-position-for-a-service-centre/ \arg\min\limits_{x_c,y_c}\sum\limits_i\sqrt{(x_i-...
• 127
1 vote
43 views

### How does gradient descent avoid local minimums?

In Neural Networks and Deep Learning, the gradient descent algorithm is described as going on the opposite direction of the gradient. Link to place in book. What prevents this strategy from landing in ...
• 115
28 views

### Understanding Gradient Descent

In Gradient Descent, we start with a random value of the co-efficient & improve upon it with each iteration. Do we move on to the next co-efficient after finding the first one, or do we work on ...
• 263
29 views

### calculating gradient descent

when using mini batch gradient descent , we perform backpropagation after each batch , ie we calculate the gradient after each batch , we also capture y-hat after each sample in the batch and finally ...
• 23
13 views

### Is it beneficial to use a batch size > 1 even when all computing power can be used?

In regards to training a neural network, it is often said that increasing the batch size decreases the network's ability to generalize, as alluded to here. This is due to the fact that training on ...
• 101
1 vote
103 views

### how to calculate loss function?

i hope you are doing well , i want to ask a question regarding loss function in a neural network i know that the loss function is calculated for each data point in the training set , and then the ...
• 23
1 vote
122 views

1 vote
23 views

### Understanding Learning Rate in depth

I am trying to understand why the learning rate does not work universally. I have two different data sets and have tested out three learning rates 0.001 ,0.01 and 0.1 . For the first data set, I was ...
• 11
15 views

### Is there a difference between AutoGrad and explicit derivatives (gradient)?

Will there be some differences between applying AutoGrad on the loss function (using a python library) and applying explicit gradient (the gradient from the paper or the update rule)? For example: ...
93 views

### Why does neural network need loss as scalar?

I have a loss function that's a weighted cross entropy loss for binary classification ...
11 views

### Is every timestep when training an LSTM part of the error, or just the end of the sequence?

I'm attempting to write my own LSTM network in C++ for fun. I've already got a regressive and classification network working with regular perceptrons and it works well. What I do currently is divide ...
• 101
1 vote
25 views

### Why it's better to use double floating point precision instead of single point precision for gradient check

I came across the following," A common pitfall is using single precision floating point to compute gradient check. It is often that case that you might get high relative errors (as high as 1e-2) ...
• 187
68 views

### Training error vs testing error for batch gradient descent on Pyrhon, can't understand what I'm supposed to do

I have this dataset containing training data and testing data , and I have to plot training and testing errors for the gradient descent algorithm with square and logistic losses. I'm a beginner, I'm a ...
19 views

### Gradient descent to minimize the sum of arc length distances on a unit sphere

I have to write a batch gradient descent algorithm to find the point on the unit sphere (in any number of dimensions) which minimizes the sum of arc length distances to each of a set of given points ...
26 views

### How does MultiVariate Linear Regression actually calculate each coefficient?

W​hile calculating the coefficients in gradient descent for multivariate linear regression, the loss accounted in all the coefficients(parameters) updating-equations is the same, loss=predicted-y x_1=...
23 views

### Loss-value of normal equation vs gradient descent

My question is if gradient descent can give a better aproximation than normal equation in Python? for the Loss function, I wrote ...
24 views

For a feedforward network or RNN, in theory we should examine the output gradients with respect to the weights over time to check whether it vanishes to zero. In my code below I am not sure whether it ...
• 113
98 views

### Adding a group specific penalty to binary cross-entropy

I want to implement a custom Keras loss function that consists of plain binary cross-entropy plus a penalty that increases the loss for false negatives from one class (each observation can belong to ...
• 1
1 vote
284 views

### MLE & Gradient Descent in Logistic Regression

In Logistic Regression, MLE is used to develop a mathematical function to estimate the model parameters, optimization techniques like Gradient Descent are used to solve this function. Can somebody ...
• 263
25 views

### Question from a paper: I do not understand why it is stated that SGD employs the bootstrapping to calculate gradient?

In this paper, they state that: As SGD employs the bootstrapping (i.e., random sampling with replacement)  for gradient calculation, we can obtain the unbiased estimation of standard gradients ...
• 121
1 vote
683 views

### Relation between MLE (Maximum Likelihood Estimation) & Gradient Descent

What are the similarities & dissimilarities between MLE (used to find the best parameters in logistic regression) & Gradient Descent?
• 263
7 views

### How to overcome extremely variable model performance?

I am training an LSTM and the model performance seems to range from near perfect to dreadful by visual inspection of predicted values - it seems by random initialisation simply retraining the model ...
249 views

### Gradient descent implementation of logistic regression

Objective Seeking for help, advise why the gradient descent implementation does not work below. Background Working on the task below to implement the logistic regression. Gradient descent Derived the ...
• 513
1 vote
13 views

### How do I find the gradients of an X -> RELU -> RELU -> Softmax network?

I am trying to use the MNIST digits dataset with logistic regression. I am only using numpy as I would like to implement a simple project myself before using something like pytorch. My network ...
• 111
1 vote
59 views

### TypeError: only size-1 arrays can be converted to Python scalars

I am doing multiple linear regression using gradient descent and am getting this type-error ...
30 views

### What stochastic gradient descent is and isn't

I've read what stochastic gradient descent is in multiple different places but everyone describes it in a slightly different way which makes me questions what is really is. Is my definition below ...
• 101
1 vote
57 views

### Why does stochastic gradient descent lead us to a minimum at all?

Why do we think that stochastic gradient descent is going to find a minimum at all? I mean on each iteration SGD moves in the direction that reduces only current batch's error (SGD doesn't care about ...
• 121