Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [gradient-descent]

The tag has no usage guidance.

0
votes
1answer
16 views

Weight training breakdown in machine learning

I'm not sure if this exists. Is there such a situation where weights in gradient descent fail to work or break up? If so, how and when?
1
vote
1answer
19 views

Where is my error in understanding gradient descent calculated two different ways?

The gradient descent algorithm is, most simply, w'(i) = w(i)-r*dC/dw(i) where w(i) are the old weights, w'(i) are the new weights, C is the cost, r is the learning rate. I'm aware of the graphical ...
1
vote
0answers
25 views

How to calculate the gradient for nce_loss in tensorflow

I need to calculate the gradient of a tensorflow that is stored. I can restore the graph and weights using: ...
2
votes
2answers
33 views

Constant Learning Rate for Gradient Decent

Given, we have a learning rate, $\alpha_n$ for the $n^{th}$ step of the gradient descent process. What would be the impact of using a constant value for $\alpha_n$ in gradient descent?
4
votes
1answer
66 views

Does Gradient Boosting detect non-linear relationships?

I wish to train some data using the the Gradient Boosting Regressor of Scikit-Learn. My questions are: 1) Is the algorithm able to capture non-linear relationships? For example, in the case of y=x^2,...
1
vote
1answer
21 views

Should I expect major performance improvements by scaling my features?

I'm trying to decide whether I should scale my features & responses for training, and I'm in a situation where I can't just try both scaling and not scaling. My features currently have an std ...
2
votes
1answer
33 views

How does Gradient Descent and Backpropagation work together?

Please forgive me as I am new to this. I have attached a diagram trying to model my understanding of neural network and Back-propagation? From videos on coursera and resources online I formed the ...
1
vote
2answers
31 views

Training Examples used in Stochastic Gradient Descent

Hi I was reading the difference between GD and SGD and found the below link. [What is the difference between Gradient Descent and Stochastic Gradient Descent? Based on this information I wanted to ...
0
votes
1answer
18 views

Confusion on Delta Rule and Error

I'm currently reading Mitchell's book for Machine Learning, and he just started gradient descent. There's one part that's really confusing me. At one point, he gives this equation for the error of a ...
3
votes
2answers
63 views

How to get out of local minimums on stochastic gradient descent?

I'm not programming a neural network but I'm looking at it from a non-hands-on, theoretical point of view and I'm currently wondering how to escape a local minimum and how to get to a global minimum. ...
0
votes
0answers
27 views

Is empirical risk the same thing as loss function?

I am reading the article Stochastic Gradient Descent Tricks by Léon Bottou (avaible here) and on the very first page they introduce empirical risk $E_n(f) = \frac{1}{n} \sum_{i=1}^{n} l(f(x_i),y_i),$ ...
1
vote
0answers
19 views

Creating a convolutional layer with weight normalization?

This paper proposes parameterizing convolutional weights by having the primary weights normalized to unit norm plus an extra scalar weight: https://arxiv.org/pdf/1602.07868.pdf Implementations do ...
0
votes
0answers
41 views

L-BFGS improvement for stochastic gradient descent

For L-BFGS method , what does it mean by the following ? At each iteration K with xk being the current iterate, it constructs the “secant” matrix Hk as a very small change (rank-2 update in linear ...
0
votes
0answers
20 views

ML / Multivariable cost minimization problems / approach summary?

General Problem Class Minimize the scalar cost function: $f(\vec r) = f(r_1, r_2, ..., r_n)$ The type of problem I'm trying to solve has many local minimum and the global minimum has not been proven ...
0
votes
1answer
27 views

How does a zero centered activation functions like tanh helps in gradient decent?

I know that, if X are all positive, or negative then the sign of the downstream gradient will be same as that of the upstream gradient, but what I don't understand is how the zero centered activation ...
1
vote
1answer
24 views

Intuition behind using the inverse of a Hessian matrix for automatically estimating the learning rate (aggression parameter) in gradient descent.

I am reviewing some course material where the lecturer suggests that instead of guessing the learning rate parameter in gradient descent implementation, one could use the inverse of the Hessian ...
2
votes
1answer
28 views

Is the magnitude of the gradient a weakness of Gradient Descent?

The formula for Gradient Descent is as follows: $$ \mathbf{w} := \mathbf{w} - \alpha\; \triangledown C $$ The gradient itself points in the direction of steepest ascent, therefore it is logical to go ...
0
votes
2answers
26 views

What does the “randomly shuffle training samples” in stochastic gradient descent attain?

What does the "randomly shuffle training samples" in stochastic gradient descent attain? I interpreted that since the training samples are used to compute $$\hat{y}=f(w^t x)$$ so if the order of $x$...
4
votes
1answer
99 views

too few data while too many degrees of freedom in linear regression

To recognize handwritten digits, I have a fully connected network, containing only 2 layers: input layer (all pixels of the image) and output layer (0 or 1). I use the simplest linear regression for ...
0
votes
0answers
19 views

Newton's method gradient and hessian formulation difficulties

Logistic regression Newton's Method Newton Method Lecture II In this picture the logistic regression cost function , Newtons Method and gradient and Hessian is defined. How to get this function that ...
1
vote
1answer
75 views

What is non-decomposable and/or non-differentiable loss function?

I have been reading some deep learning literature and came up with these concepts of non-decomposable and non-differentiable loss functions. My question is are these same thing? if not how are they ...
0
votes
0answers
20 views

why don't SOTA neural network models back propagation use conjugate gradients?

My understanding is that the conjugate gradient method is faster than gradient descent because it does less zig zags while descending. How come the state of the art papers I see all use gradient ...
0
votes
1answer
128 views

Should the minimum value of a cost (loss) function be equal to zero?

We know optimization techniques search in the space of all the possible parameters for a parameter set that minimizes the cost function of the model. The most well-known loss functions, like MSE or ...
0
votes
1answer
93 views

Gradient Descent Python Implementation isnt converging

I trying to implement gradient descent in Python and I am following andrew ng course in order to follow the math. However, my implementation isnt working as I expect it to. It would be great you the ...
0
votes
1answer
46 views

How to Define a Cost Fucntion?

I want to define a cost function in python to identify optimum value in days when i should end a marketing campaign to save spend on campaigns not generating traffic good traffic. Problem is I dont ...
1
vote
0answers
12 views

How to optimize the separation of two distributions from binary classfication

Given a sample where for each individual a classification is predetermined (e.g. sick or not) and 5 random variables are measured. The random variables are on the same scale but from differnt bins. E....
2
votes
4answers
52 views

Is gradient descent slower for finite differences?

In gradient descent, we updated each parameter $\theta_i$ in the direction which minimizes a function $f(\theta_1,\theta_2,\dots,\theta_N)$ by doing $$\theta_1 \leftarrow \theta_1 - \alpha \frac{\...
4
votes
2answers
80 views

Why Gradient methods work in finding the parameters in Neural Networks?

After reading quite a lot of papers (20-30 or so), I feel that I am quite not understanding things. Let us focus on the supervised learnings (for example). Given a set of data $\mathcal{D}_{train}=\{...
0
votes
0answers
69 views

What are the reasons of select a optimizer to be SGD or Adam in DQN?Why?

I saw several comparison between SGD, RMSPROP and ADAM but what I am looking for is their comparsion in DQN algorithm? What is best to select as optimizer SGD or Adam in DQN? Why? Please check the ...
1
vote
0answers
62 views

Why perceptron does not converge on data not linearly separable

This is how I understand the perceptron algorithm. The perceptron loss function is the hinge loss $\ell(w,x,y) = \max(0, -yw\cdot x)$. Suppose the data set is $D = \{(x_1,y_1),\dots,(x_n,y_n)\}$ with ...
1
vote
1answer
215 views

Difference between Gradient Descent and Normal Equation in Linear Regression

Hi I am new to Linear Regression. I want to know what is the difference b/w Gradient Descent and Mean Square Error in Linear Regression using machine learning? And When to use Gradient ...
2
votes
0answers
78 views

clipping the reward for adam optimizer in keras

I would like to clip the reward in keras. I saw it is possible to clip the norm and clip the value is sgd as follows: ...
1
vote
0answers
20 views

What is the significance of underflow during parameter update using stochastic gradient descent?

Background I am using scikit-learn's MLPRegressor to learn a model with the following arguments: ...
2
votes
1answer
76 views

multilayer perceptron do not converge

I have been coding my own multi layer perceptron in MATLAB and it can be compiled without error. My training data features,x, has values from 1 to 360, and training data output, y, has the value of ...
2
votes
1answer
71 views

Backpropagation

I use chain rule when doing backpropagation and then I do Gradient Descent with weighting coefficient and I am updating the weight, so I do not understand how the method works in the equations below. ...
1
vote
1answer
1k views

What is the difference between SGD classifier and the Logisitc regression?

To my understanding, the SGD classifier, and Logistic regression seems similar. An SGD classifier with loss = 'log' implements Logistic regression and loss = 'hinge' implements Linear SVM. I also ...
1
vote
0answers
32 views

RMSProp Optimizer Performing Poorly

I am building an RNN and have decided to try RMSProp as an alternative to sgd. Here is my implementation: ...
0
votes
0answers
48 views
2
votes
1answer
34 views

Why are optimization algorithms slower at critical points?

I just found the animation below from Alec Radford's presentation: As visible, all algorithms are considerably slowed down at saddle point (where derivative is 0) and quicken up once they get out of ...
0
votes
0answers
20 views

Why my Simple DL model with default SGD working very poorly in keras?

When I'm answering questions in StackOverflow, I found this question about the poor performance of DL model. I tried to implement the same code in my machine to see whats happening. I have also ...
3
votes
0answers
86 views

Explanations about ADAM Optimizer algorithm

I'm a beginner in Machine learning and i'm searching for some optimizer for the gradient descent. I've searched many topics about that, and did a state of art of all these optimizers. I have just one ...
0
votes
1answer
28 views

Differentiating roadmap of a loss function

Let's say I'm performing Stochastic Gradient Descent (SGD) on binary cross entropy error while optimizing weight $w_{2}$. Binary cross entropy error: $$L(y|p(x_{i}))=-y_{i}*ln(p(x_{i}))-(1-y_{i})*...
2
votes
1answer
47 views

Gradient descent multidimensional linear regression - does learning rate affects concurrency? [closed]

I wonder if gradient descent for multidimensional regression always finds the right result? I feel like this doesn't always have to be true. I have done some calculations and actually got correct ...
7
votes
3answers
5k views

What is the difference between Gradient Descent and Stochastic Gradient Descent?

What is the difference between Gradient Descent and Stochastic Gradient Descent? I do not know these things very well, can you describe it with a short example? Good luck with, Engin
0
votes
1answer
41 views

Algorithm for backpropagation through time

I am reading through this article trying to understand the bptt algorithm, in the context of an RNN. However there is one part I don’t understand: ...
0
votes
1answer
22 views

What is the best statistical measure tool to measure how close data is to fitted regression line if outliers are not fitted

I am using a custom algorithm based on Gradient descent which computes the best fit on a training dataset. In this data set I have outliers i.e. data points that I do not want to fit. The algorithm is ...
0
votes
0answers
156 views

Machine Learning: Stochastic gradient descent method for logistic regression in R

I am trying to write a code to solve the following problem (As stated in HW5 in the CalTech course Learning from Data): In this problem you will create your own target function f (probability in this ...
1
vote
1answer
42 views

What is the intuition behind Ridge Regression and Adapting Gradient Descent algorithms?

So I was going through Adaptive Gradient Descent, and learning the intuition behind it: optimizing the learning algorithm, and getting the model to converge faster. The way AdaGrad does this, is by ...
1
vote
2answers
693 views

How the combination of cross entropy loss and gradient descent penalizes and rewards

For a simple problem of classification (C classes) using the softmax classifier, most people use the cross-entropy loss function to quantify the objective. The cross-entropy loss is: $$L = -\sum_{i=1}...