Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [gradient-descent]

Gradient Descent is an algorithm for finding the minimum of a function. It iteratively calculates partial derivatives (gradients) of the function and descends in steps proportional to those partial derivatives. One major application of Gradient Descent is fitting a parameterized model to a set of data: the function to be minimized is an error function for the model.

1
vote
2answers
125 views

which metric is better for boosting methods

I work on a dataset of 300 000 samples and I try to make a comparison between logistic regression (with gradients descent) and a LightBoost for binary classification in order to choose the better one. ...
0
votes
0answers
26 views

Perceptron learning algorithm. Gradient descent

Linear classifier: $a(x,w) = sign\langle\,x,w\rangle$, if sign is positive object belongs to $+1$ class and if it is negative to $-1$. When we use gradient descent to train the perceptron, we are ...
0
votes
0answers
10 views

How to retain dependency between variables in PyTorch?

I am modeling k-dimensional positions over time t = 0...T using a set of initial positions Z0 with requires_grad=True and storing the results in Z with requires_grad=False for the remaining T-1 time ...
0
votes
0answers
28 views

Vanishing gradient problem for recent stochastic recurrent neural networks

Recently, I've found some papers about generative recurrent models. All have attached sub-networks like prior/encoder/decoder/etc. to well-known LSTM cell for composing an aggregation of new-type RNN ...
1
vote
1answer
28 views

Neural network back propagation gradient descent calculus

So I've drawn a neural network diagram below: where $x_1, x_2,\ldots,x_m$ are the input layer, $h_1, h_2$ are the hidden layer and $\hat y_1, \hat y_2,\ldots \hat y_k$ are the output layer. In the $W^...
1
vote
1answer
22 views

Gradient descent with infinite gradient value

Given a function $f(x)$ and $\frac{\partial f(x)}{\partial x_i}=\frac{f^2(x1,...,x_i+\pi/2,...,x_n)-f^2(x1,...,x_i-\pi/2,...,x_n)}{f(x)}$. When $f(x)\to0$, $\frac{\partial f(x)}{\partial x_i}$ could ...
0
votes
0answers
10 views

How to check If Stochastic Gradient Descent produces the optimum MSE for linear regression

I am implementing SGD in linear regression . By varying the learning rate and sample size different weight vectors are produced. These produce different MSE that are far apart. Is it possible to ...
1
vote
2answers
25 views

Gradient Descent Convergence

I'm a double major in Math and CS interested in Machine Learning. I'm currently taking the popular Coursera course by Prof. Andrew. He's talking and explaining Gradient Descent but I can't avoid ...
0
votes
0answers
30 views

Gradient computation in neural networks

I am working on understanding gradient computation in neural networks. But there is an issue with my computation. Let the weights between input (X) and hidden layer between $W_{ij}$ and hidden layer ...
0
votes
0answers
27 views

Logistic regression gradient descent classifier - more iterations leads to worse accuracy

I'm running a binary classification logistic regression. When I run gradient descent for 100 iterations I get ~ 90% prediction accuracy (cost function is decreasing constantly but hasn't converged yet)...
2
votes
2answers
44 views

Calculating derivative of error at point x with respects to weight w_j

I don't know how the equation below goes from line 2 to 3 after the derivative term is moved inside the brackets. Specifically, how is it calculating the derivative of log(y_hat)? Also, if anyone ...
1
vote
1answer
14 views

Partial derviative of prediction (sigmoid applied) with respect to weight

I am very confused as to where a seemingly "extra" term is included in the above mentioned calculation in my Udacity course. The above is taking the derivative of a sigmoid so why isn't it just $$...
1
vote
1answer
59 views

Understanding general approach to updating optimization function parameters

This question not related to a specific method or technique, rather there is a broader concept that I'm struggling to see clearly. Introduction In machine learning, we have loss functions that we're ...
12
votes
4answers
3k views

Is Gradient Descent central to every optimizer?

I want to know whether Gradient descent is the main algorithm used in optimizers like Adam, Adagrad, RMSProp and several other optimizers.
1
vote
1answer
47 views

Linear classifier and gradient descent

I understood that gradient descent is needed to find the local extremum of any function, but how is it applied to the linear classifier (single matrix, two classes case, for example)? How does it step ...
0
votes
0answers
10 views

Nesterov Momentum update equation

I am currently taking CS231n. In the lecture, training neural networks part 2, Nesterov momentum is introduced where I understood theoretically that the gradient is calculated at a later point where ...
0
votes
1answer
17 views

Can we use decreasing step size to replace mini-batch in SGD?

As far as I know, mini-batch can be used to reduce the variance of the gradient, but I am also considering if we can achieve the same result if we use the decreasing step size and only single sample ...
0
votes
0answers
17 views

Do 3D-CNN suffer from exploding and vanishing gradient?

I am building a video classification network, I see 2 options for the same CNN+LSTM and 3D-CNNs, Do 3D-CNNs suffers from exploding and vanishing gradients like LSTMs or are 3d-CNN better at handling ...
1
vote
0answers
24 views

Modelling a startup's funding journey with Brownian Motion

I am trying to implement a "light" version of a paper (Hunter, Saini & Saman 2017), in which the authors build a model capable of predicting the probability that a startup will exit (either by ...
0
votes
2answers
28 views

Weight training breakdown in machine learning

I'm not sure if this exists. Is there such a situation where weights in gradient descent fail to work or break up? If so, how and when?
1
vote
1answer
25 views

Where is my error in understanding gradient descent calculated two different ways?

The gradient descent algorithm is, most simply, w'(i) = w(i)-r*dC/dw(i) where w(i) are the old weights, w'(i) are the new weights, C is the cost, r is the learning rate. I'm aware of the graphical ...
1
vote
0answers
36 views

How to calculate the gradient for nce_loss in tensorflow

I need to calculate the gradient of a tensorflow that is stored. I can restore the graph and weights using: ...
2
votes
2answers
47 views

Constant Learning Rate for Gradient Decent

Given, we have a learning rate, $\alpha_n$ for the $n^{th}$ step of the gradient descent process. What would be the impact of using a constant value for $\alpha_n$ in gradient descent?
4
votes
1answer
84 views

Does Gradient Boosting detect non-linear relationships?

I wish to train some data using the the Gradient Boosting Regressor of Scikit-Learn. My questions are: 1) Is the algorithm able to capture non-linear relationships? For example, in the case of y=x^2,...
1
vote
1answer
26 views

Should I expect major performance improvements by scaling my features?

I'm trying to decide whether I should scale my features & responses for training, and I'm in a situation where I can't just try both scaling and not scaling. My features currently have an std ...
2
votes
1answer
52 views

How does Gradient Descent and Backpropagation work together?

Please forgive me as I am new to this. I have attached a diagram trying to model my understanding of neural network and Back-propagation? From videos on coursera and resources online I formed the ...
1
vote
2answers
35 views

Training Examples used in Stochastic Gradient Descent

Hi I was reading the difference between GD and SGD and found the below link. [What is the difference between Gradient Descent and Stochastic Gradient Descent? Based on this information I wanted to ...
0
votes
1answer
22 views

Confusion on Delta Rule and Error

I'm currently reading Mitchell's book for Machine Learning, and he just started gradient descent. There's one part that's really confusing me. At one point, he gives this equation for the error of a ...
3
votes
2answers
152 views

How to get out of local minimums on stochastic gradient descent?

I'm not programming a neural network but I'm looking at it from a non-hands-on, theoretical point of view and I'm currently wondering how to escape a local minimum and how to get to a global minimum. ...
0
votes
0answers
35 views

Is empirical risk the same thing as loss function?

I am reading the article Stochastic Gradient Descent Tricks by Léon Bottou (avaible here) and on the very first page they introduce empirical risk $E_n(f) = \frac{1}{n} \sum_{i=1}^{n} l(f(x_i),y_i),$ ...
1
vote
0answers
23 views

Creating a convolutional layer with weight normalization?

This paper proposes parameterizing convolutional weights by having the primary weights normalized to unit norm plus an extra scalar weight: https://arxiv.org/pdf/1602.07868.pdf Implementations do ...
0
votes
1answer
30 views

How does a zero centered activation functions like tanh helps in gradient decent?

I know that, if X are all positive, or negative then the sign of the downstream gradient will be same as that of the upstream gradient, but what I don't understand is how the zero centered activation ...
1
vote
1answer
35 views

Intuition behind using the inverse of a Hessian matrix for automatically estimating the learning rate (aggression parameter) in gradient descent.

I am reviewing some course material where the lecturer suggests that instead of guessing the learning rate parameter in gradient descent implementation, one could use the inverse of the Hessian ...
3
votes
1answer
35 views

Is the magnitude of the gradient a weakness of Gradient Descent?

The formula for Gradient Descent is as follows: $$ \mathbf{w} := \mathbf{w} - \alpha\; \triangledown C $$ The gradient itself points in the direction of steepest ascent, therefore it is logical to go ...
0
votes
2answers
43 views

What does the “randomly shuffle training samples” in stochastic gradient descent attain?

What does the "randomly shuffle training samples" in stochastic gradient descent attain? I interpreted that since the training samples are used to compute $$\hat{y}=f(w^t x)$$ so if the order of $x$...
4
votes
1answer
110 views

too few data while too many degrees of freedom in linear regression

To recognize handwritten digits, I have a fully connected network, containing only 2 layers: input layer (all pixels of the image) and output layer (0 or 1). I use the simplest linear regression for ...
1
vote
1answer
129 views

What is non-decomposable and/or non-differentiable loss function?

I have been reading some deep learning literature and came up with these concepts of non-decomposable and non-differentiable loss functions. My question is are these same thing? if not how are they ...
0
votes
1answer
261 views

Should the minimum value of a cost (loss) function be equal to zero?

We know optimization techniques search in the space of all the possible parameters for a parameter set that minimizes the cost function of the model. The most well-known loss functions, like MSE or ...
0
votes
1answer
205 views

Gradient Descent Python Implementation isnt converging

I trying to implement gradient descent in Python and I am following andrew ng course in order to follow the math. However, my implementation isnt working as I expect it to. It would be great you the ...
0
votes
1answer
50 views

How to Define a Cost Fucntion?

I want to define a cost function in python to identify optimum value in days when i should end a marketing campaign to save spend on campaigns not generating traffic good traffic. Problem is I dont ...
1
vote
0answers
13 views

How to optimize the separation of two distributions from binary classfication

Given a sample where for each individual a classification is predetermined (e.g. sick or not) and 5 random variables are measured. The random variables are on the same scale but from differnt bins. E....
3
votes
4answers
59 views

Is gradient descent slower for finite differences?

In gradient descent, we updated each parameter $\theta_i$ in the direction which minimizes a function $f(\theta_1,\theta_2,\dots,\theta_N)$ by doing $$\theta_1 \leftarrow \theta_1 - \alpha \frac{\...
4
votes
2answers
88 views

Why Gradient methods work in finding the parameters in Neural Networks?

After reading quite a lot of papers (20-30 or so), I feel that I am quite not understanding things. Let us focus on the supervised learnings (for example). Given a set of data $\mathcal{D}_{train}=\{...
0
votes
0answers
105 views

What are the reasons of select a optimizer to be SGD or Adam in DQN?Why?

I saw several comparison between SGD, RMSPROP and ADAM but what I am looking for is their comparsion in DQN algorithm? What is best to select as optimizer SGD or Adam in DQN? Why? Please check the ...
1
vote
0answers
88 views

Why perceptron does not converge on data not linearly separable

This is how I understand the perceptron algorithm. The perceptron loss function is the hinge loss $\ell(w,x,y) = \max(0, -yw\cdot x)$. Suppose the data set is $D = \{(x_1,y_1),\dots,(x_n,y_n)\}$ with ...
1
vote
1answer
281 views

Difference between Gradient Descent and Normal Equation in Linear Regression

Hi I am new to Linear Regression. I want to know what is the difference b/w Gradient Descent and Mean Square Error in Linear Regression using machine learning? And When to use Gradient ...
2
votes
0answers
182 views

clipping the reward for adam optimizer in keras

I would like to clip the reward in keras. I saw it is possible to clip the norm and clip the value is sgd as follows: ...
1
vote
0answers
33 views

What is the significance of underflow during parameter update using stochastic gradient descent?

Background I am using scikit-learn's MLPRegressor to learn a model with the following arguments: ...