Questions tagged [gradient-descent]
Gradient Descent is an algorithm for finding the minimum of a function. It iteratively calculates partial derivatives (gradients) of the function and descends in steps proportional to those partial derivatives. One major application of Gradient Descent is fitting a parameterized model to a set of data: the function to be minimized is an error function for the model.
444
questions
0
votes
1
answer
12
views
moments of weight vectors in Adam
When performing backpropagation with Adam algorithm, are the moment and the second moment of the weight vectors calculated also for the weights in hidden layers?
- 122
0
votes
0
answers
9
views
Practical limits of backpropagation
I have been performing a few neural network algorithms to perform linear regression.
First, I tried SGD. It takes around 20000 epochs to converge, but loss (MSE) is around 0.001.
(To note that perfect ...
- 122
0
votes
1
answer
17
views
what is static objective function
Paper on Adam mentions Stationary Objective function. I am not able to find its definition on Internet (or may be it's there with some other name and I am not able to figure it out). I will be ...
- 203
0
votes
1
answer
12
views
AdaGrad denominator exponents comparison
Adagrad adapts the learning rate $ \alpha $ all along the gradient descent process, by dividing each weight on a quantity based on the sum of the previous squared gradient up to time $ t $. Therefore, ...
1
vote
0
answers
21
views
How to calculate gradient of MSE in backpropagation? [duplicate]
I want to implement a neural network from scratch to solve linear regression by using backpropagation. I don't understand how to compute the gradient of the MSE cost function with respect to each ...
- 122
-1
votes
0
answers
13
views
How can I tweak hyper parameters to quickly achieve 0 loss?
I am training on whole dataset. That is there are 2048 training examples and mini batch is also of size 2048 samples. So this is sort of batch gradient descent.
I had following two training runs:
Run ...
- 99
0
votes
0
answers
7
views
Batchnorm and gradient attenuation
Am I understood right that if we put batchnorm layer after every conv layer and our network is very deep, then we won't have a problem of gradient attenuation? In other words, is it true that ...
- 101
0
votes
0
answers
11
views
Understanding gradient of skip gram
I am trying to understand gradient calculation for skip gram with softmax output and cross entropy loss.
I am referring these articles: 1, 2, 3.
The all calculate the error as follows:
$$E=-\sum_{c=1}...
- 99
0
votes
0
answers
8
views
Polynomial Regression coefficient extraction after data normalisation for Mini-Batch SGD
I've written python function that uses a stochastic mini-batch algorithm to compute the optimal polynomial coefficients for a given degree $m$, however this involved normalising the data where
$$
x' = ...
- 101
0
votes
1
answer
26
views
Why is cross entropy loss averaged and not used directly as a sum during model training(such as in neural networks)
Why is the cross entropy loss for all training examples(or the training examples in a batch) averaged over size of the training set(or batch size) ?
Why is it not just summed and used ?
1
vote
1
answer
16
views
How train a pre-trained model based on new dataset?
I have trained a deep nn model based on some existing data. In the meantime, I have collected more data and label them so that I can feed it to the model to improve its performance. The questions is, ...
- 253
0
votes
0
answers
15
views
How to derive expression for gradient in BPPT
I have the following problem: I am trying to derive final expressions for error gradients in a simple recurrent neural network (Backpropagation through Time, BPPT). The parameters and state update ...
0
votes
1
answer
35
views
GAN Generator Backpropagation Gradient Shape Doesn't Match
In the TensorFlow example (https://www.tensorflow.org/tutorials/generative/dcgan#the_discriminator) the discriminator has a single output neuron (assume batch_size=1).
Then over in the training loop ...
- 3
0
votes
0
answers
35
views
Understanding the stochastic average gradient (SAG) algorithm used in sklearn
For pedagogical purposes I've been trying to create my own implementation of the stochastic average gradient (SAG) algorithm in a logistic regression framework. Page 10 of the associated paper ...
0
votes
0
answers
7
views
What is the meaning of uncentered variance?
I read some articles that describe ADAM optimizer and they used uncentered variance expression.
I'm not sure I understand the ...
- 363
0
votes
1
answer
28
views
What does the ellipse in Gradient descent describes?
I saw multiple articles describes GD or SGD with the following diagram:
I didn't saw any explanation about the ellipses.
What ...
- 363
0
votes
0
answers
7
views
In RMSProp how squares of the gradients dumps out the oscillations?
I read multiple articles that say that RMSProp use the squares of the gradients in order to dump oscillations.
It's not intuitive to me how the squares of the ...
- 363
1
vote
0
answers
33
views
How Adam optimizer influence the learning rate? [closed]
I read some papers about how ADAM optimizer works, and there are some issues which seems that are confusing:
ADAM equations are:
...
- 363
0
votes
1
answer
29
views
What will happen if we apply Gradient Ascent?
I have built a simple neural network on MNIST, but instead of moving toward the opposite direction of gradients, I moved in the same direction of it just by applying( pytorch ):
...
- 253
1
vote
1
answer
67
views
Does settings $\beta_1 = 0$ or $\beta_2 = 0$ means that ADAM behaves as RMSprop or Momentum?
I read on ADAM optimizer, and I saw multiple quotes which say that ADAM is a combination of Momentum and RMSprop optimizers.
So if we:
Set $\beta_1 = 0$ does it means that ADAM behaves exactly as ...
- 363
0
votes
1
answer
33
views
Normalising data for simple linear regression
Consider a simple linear regression problem where:
X = [1,2,3,4,5,100,200]
Y= [2,4,6,8,10,200,400]
Clearly, the relationship is of the form $y=2x$; While trying ...
- 13
0
votes
0
answers
8
views
How backward() is calculated in CrossEntropyLoss?
I have a simple Linear model and I need to calculate the loss for it. I applied two CrossEntropyLoss and NLLLoss but I want to ...
- 253
6
votes
4
answers
1k
views
How to fit a math formula to data?
I have math formula and some data and I need to fit the data to this model.
The math is like. $y(x) = ax^k + b$ and I need to estimate the $a$ and the $b$.
I have triet gradient descend to estimate ...
- 253
0
votes
1
answer
94
views
Why does Adam outperform SGD in logistic regression?
I am training a logistic regression model. In case it matters, the features are 1376-dimensional embeddings output from a neural network. I tried both SGD and Adam with a learning rate of $10^{-3}$ ...
- 113
0
votes
1
answer
58
views
Difference between sklearn's LogisticRegression and SGDClassifier?
What is the difference between sklearn's LogisticRegression classifier and its SGDClassifier? I understand that the SGD is an ...
0
votes
0
answers
38
views
Why backpropagation is done in every epoch when loss is always scalar?
I understand the backpropagation algorithm that it calculates the derivate of loss with respect to all the parameters in the neural network. My question is this derivate is constant right because the ...
- 101
0
votes
0
answers
22
views
Question about input value in Gradient descent
I am currently going through Udacity´s online course "Intro to Deep Learning with PyTorch". In one of the videos covering the Gradient descent algorithm they show the formula for how the ...
- 1
0
votes
1
answer
66
views
Setting shape of RaggedTensor with Known Shape
I'm working with RaggedTensors to manipulate a dense tensor. Something like this :
...
- 53
2
votes
0
answers
28
views
Can I find the input that maximises the output of a Neural Network?
So I trained a 2 layer Neural Network for a regression problem that takes $D$ features $(x_1,...,x_D)$ and outputs a real value $y$. With the model already trained (weights optimised, fixed), can I ...
0
votes
0
answers
55
views
XGBoost Gradient Boosting and Gradient Descent confusion
I am trying to understand how the XGBoost determines the next tree. Some sources state that the model uses gradient descent to find the optimal option:
This answer from a question on this s.e. also ...
- 1
0
votes
0
answers
16
views
compare decision tree vs extended gradient boosting mathematically?
If we want to compare decision tree vs extended gradient boosting vs xgboost mathematically, what are their differences?
- 1,754
0
votes
2
answers
40
views
Can Gradient Descent boosted decision trees miss the forest for the trees?
My understanding of this stuff is pretty basic so my semantics may be off, but bare with me.
XGBoost and other gradient descent packages make the best possible split of the data right off the bat. ...
- 113
0
votes
1
answer
27
views
How to Improve MLP ANN accuracy
I am trying to improve the accuracy of my model over the UCI Breast Cancer Dataset. There's 426 records, and it is a binary classification model.
...
- 1
0
votes
0
answers
21
views
Why does merging all loss in a batch make sense?
I built a vanilla GRU net to resolve a binary classification problem, and train it with a batch size greater than 1.
Let's say my NN model got a correct result with sample-0, and got a incorrect ...
- 101
0
votes
0
answers
21
views
Does the loss function in a deep neural network act as a norm?
I read somewhere that the Mean squared error loss function acts as L2 norm of the paramter vector. I would like to know if I am using binary cross entropy loss function, do I need to calculate the ...
- 101
0
votes
0
answers
21
views
Why is this an incorrect update of the parameters in the gradient descent algorithm? (Bishop, Pattern Recognition and Machine Learning)
Let's say we are performing a linear regression, with general model $y(x,w) = w_0 + w_1x$. The error function is $E(w) = \frac{1}{2N}\sum_n ((y(x_n,w)-t_n)^2$, for $N$ datapoints ${(x_n,t_n)}$ (...
0
votes
1
answer
40
views
AdaGrad: Intuition
The update formula for Adagrad is:
\begin{equation}
w^i(t)=w^i(t-1) -\frac{\eta}{\sqrt{\epsilon +\sum_{1}^t |\nabla_i\mathcal{L}}|^2} \nabla_i\mathcal{L}
\end{equation}
It indicates that if the ...
- 21
3
votes
1
answer
58
views
Why is it an advantage "that Markov chains are never needed" to obtain gradients?
In the original GAN (Generative Adversarial Network) paper, Generative adversarial networks by I. Goodfellow, J. Pouget-Abadie, M. Mirza et. al. they state an advantage of the GAN is "that Markov ...
- 33
0
votes
0
answers
177
views
Proof that averaging weights is equal to averaging gradients (FedSGD vs FedAvg)
The first paper of Federated Learning "Communication-Efficient Learning of Deep Networks from Decentralized Data" presents FedSGD and FedAvg. In Federated Learning the learning task is ...
- 221
2
votes
2
answers
470
views
SGD performing better than Adam in Random minority oversampling, I don't know what is the reason. Help
So my dataset image before and after balancing looks like this:
But when I train with Adam(0.0001) and SGD(0.0001), the results are very different. Why? What is going on under the hood?
This is ...
- 21
0
votes
1
answer
132
views
Gradient descent vs stochastic gradient descent vs mini-batch gradient descent with respect to working step/example
I am trying to understand the working of gradient descent, stochastic gradient descent and mini-batch gradient descent.
In case of gradient descent, gradient is computed on the entire dataset at each ...
- 187
0
votes
1
answer
37
views
Is gradient descent useful to get the least mean squared error in linear regression?
I am new to machine learning.
I have read about the linear regression where-in the ideal model is a line which has the least mean squared error. In multi-variable linear regression we would have a ...
- 187
0
votes
0
answers
28
views
Gradient descent/Adam converging to suboptimal solutions
I am using neural nets to find the minimum of a complex function to which I compute the mean (crit in my code). Here is my net :
...
- 1
0
votes
1
answer
32
views
why we have problem with gradients when feature values are of different range?
A blog below mentioned.
" Because different features do not have similar ranges of values, gradients may take a long time, oscillate back and forth, and take a long time before they can finally ...
- 11
0
votes
0
answers
86
views
calculating derivative of bias in backpropagation
Looking at the algorithm in wikipedia, we can implement backpropagation by calculating:
$$\delta^{L}=\left(f^{L}\right)'\cdot\nabla_{a^{L}}C$$
(where I treat $\left(f^{L}\right)'$ as an $n\times n$ ...
- 101
1
vote
2
answers
70
views
In Gradient descent, Why the gradient of cost function do not have to be normalized into unit vector
From my background, I understand that the purpose of having a learning rate (α) is to normalize the magnitude of gradient (▽J), so the step size can properly converge the local minima
Since α is ...
- 11
0
votes
0
answers
52
views
Why does the sigmoid function being not centered on zero means that the gradient will always be the same sign with positive inputs?
In my course I have this affirmation but I don't see why. It is referring to neural networks' gradient descent.
- 33
0
votes
1
answer
56
views
Understanding Conjugate Gradient Optimization methods
As a beginner in ML, I find it hard to understand how Conjugate Gradient Optimization methods work. The sources I've looked up online have a very complicated explanation.
Can someone explain in a ...
- 273
0
votes
0
answers
13
views
How to calculate the expression for the gradient of softmax + cross entropy with respect to weights?
I'm learning cs231n on my own.
The Softmax classifier has the following loss function:
to make this clear:
$L_i$ is the loss for a particular training input
$f_j$ is the $j$th element of the vector, ...
- 115
0
votes
1
answer
128
views
Why would we add regularization loss to the gradient itself in an SVM?
I'm doing CS 231n on my own. I'm looking at this solution to a question that implements a SVM.
Relevant code:
...
- 115