Questions tagged [gradient-descent]

Gradient Descent is an algorithm for finding the minimum of a function. It iteratively calculates partial derivatives (gradients) of the function and descends in steps proportional to those partial derivatives. One major application of Gradient Descent is fitting a parameterized model to a set of data: the function to be minimized is an error function for the model.

Filter by
Sorted by
Tagged with
1
vote
3answers
229 views

Gradient Descent Convergence

I'm a double major in Math and CS interested in Machine Learning. I'm currently taking the popular Coursera course by Prof. Andrew. He's talking and explaining Gradient Descent but I can't avoid ...
2
votes
2answers
77 views

Calculating derivative of error at point x with respects to weight w_j

I don't know how the equation below goes from line 2 to 3 after the derivative term is moved inside the brackets. Specifically, how is it calculating the derivative of log(y_hat)? Also, if anyone ...
1
vote
1answer
35 views

Partial derviative of prediction (sigmoid applied) with respect to weight

I am very confused as to where a seemingly "extra" term is included in the above mentioned calculation in my Udacity course. The above is taking the derivative of a sigmoid so why isn't it just $$...
1
vote
1answer
84 views

Understanding general approach to updating optimization function parameters

This question not related to a specific method or technique, rather there is a broader concept that I'm struggling to see clearly. Introduction In machine learning, we have loss functions that we're ...
13
votes
4answers
3k views

Is Gradient Descent central to every optimizer?

I want to know whether Gradient descent is the main algorithm used in optimizers like Adam, Adagrad, RMSProp and several other optimizers.
1
vote
1answer
307 views

Linear classifier and gradient descent

I understood that gradient descent is needed to find the local extremum of any function, but how is it applied to the linear classifier (single matrix, two classes case, for example)? How does it step ...
0
votes
2answers
61 views

Can we use decreasing step size to replace mini-batch in SGD?

As far as I know, mini-batch can be used to reduce the variance of the gradient, but I am also considering if we can achieve the same result if we use the decreasing step size and only single sample ...
1
vote
0answers
29 views

Modelling a startup's funding journey with Brownian Motion

I am trying to implement a "light" version of a paper (Hunter, Saini & Saman 2017), in which the authors build a model capable of predicting the probability that a startup will exit (either by ...
0
votes
2answers
50 views

Weight training breakdown in machine learning

I'm not sure if this exists. Is there such a situation where weights in gradient descent fail to work or break up? If so, how and when?
1
vote
1answer
35 views

Where is my error in understanding gradient descent calculated two different ways?

The gradient descent algorithm is, most simply, w'(i) = w(i)-r*dC/dw(i) where w(i) are the old weights, w'(i) are the new weights, C is the cost, r is the learning rate. I'm aware of the graphical ...
1
vote
0answers
147 views

How to calculate the gradient for nce_loss in tensorflow

I need to calculate the gradient of a tensorflow that is stored. I can restore the graph and weights using: ...
2
votes
2answers
146 views

Constant Learning Rate for Gradient Decent

Given, we have a learning rate, $\alpha_n$ for the $n^{th}$ step of the gradient descent process. What would be the impact of using a constant value for $\alpha_n$ in gradient descent?
4
votes
1answer
510 views

Does Gradient Boosting detect non-linear relationships?

I wish to train some data using the the Gradient Boosting Regressor of Scikit-Learn. My questions are: 1) Is the algorithm able to capture non-linear relationships? For example, in the case of y=x^2,...
1
vote
1answer
33 views

Should I expect major performance improvements by scaling my features?

I'm trying to decide whether I should scale my features & responses for training, and I'm in a situation where I can't just try both scaling and not scaling. My features currently have an std ...
1
vote
1answer
100 views
6
votes
1answer
6k views

How does Gradient Descent and Backpropagation work together?

Please forgive me as I am new to this. I have attached a diagram trying to model my understanding of neural network and Back-propagation? From videos on Coursera and resources online I formed the ...
1
vote
2answers
99 views

Training Examples used in Stochastic Gradient Descent

Hi I was reading the difference between GD and SGD and found the below link. [What is the difference between Gradient Descent and Stochastic Gradient Descent? Based on this information I wanted to ...
0
votes
1answer
108 views

Confusion on Delta Rule and Error

I'm currently reading Mitchell's book for Machine Learning, and he just started gradient descent. There's one part that's really confusing me. At one point, he gives this equation for the error of a ...
5
votes
2answers
4k views

How to get out of local minimums on stochastic gradient descent?

I'm not programming a neural network but I'm looking at it from a non-hands-on, theoretical point of view and I'm currently wondering how to escape a local minimum and how to get to a global minimum. ...
4
votes
2answers
1k views

Is empirical risk the same thing as loss function?

I am reading the article Stochastic Gradient Descent Tricks by Léon Bottou (avaible here) and on the very first page they introduce empirical risk $E_n(f) = \frac{1}{n} \sum_{i=1}^{n} l(f(x_i),y_i),$ ...
1
vote
0answers
69 views

Creating a convolutional layer with weight normalization?

This paper proposes parameterizing convolutional weights by having the primary weights normalized to unit norm plus an extra scalar weight: https://arxiv.org/pdf/1602.07868.pdf Implementations do ...
1
vote
2answers
72 views

How does a zero centered activation functions like tanh helps in gradient decent?

I know that, if X are all positive, or negative then the sign of the downstream gradient will be same as that of the upstream gradient, but what I don't understand is how the zero centered activation ...
1
vote
1answer
144 views

Intuition behind using the inverse of a Hessian matrix for automatically estimating the learning rate (aggression parameter) in gradient descent.

I am reviewing some course material where the lecturer suggests that instead of guessing the learning rate parameter in gradient descent implementation, one could use the inverse of the Hessian ...
2
votes
1answer
140 views

Is the magnitude of the gradient a weakness of Gradient Descent?

The formula for Gradient Descent is as follows: $$ \mathbf{w} := \mathbf{w} - \alpha\; \triangledown C $$ The gradient itself points in the direction of steepest ascent, therefore it is logical to go ...
0
votes
2answers
214 views

What does the “randomly shuffle training samples” in stochastic gradient descent attain?

What does the "randomly shuffle training samples" in stochastic gradient descent attain? I interpreted that since the training samples are used to compute $$\hat{y}=f(w^t x)$$ so if the order of $x$...
4
votes
1answer
133 views

too few data while too many degrees of freedom in linear regression

To recognize handwritten digits, I have a fully connected network, containing only 2 layers: input layer (all pixels of the image) and output layer (0 or 1). I use the simplest linear regression for ...
1
vote
1answer
808 views

What is non-decomposable and/or non-differentiable loss function?

I have been reading some deep learning literature and came up with these concepts of non-decomposable and non-differentiable loss functions. My question is are these same thing? if not how are they ...
0
votes
1answer
2k views

Should the minimum value of a cost (loss) function be equal to zero?

We know optimization techniques search in the space of all the possible parameters for a parameter set that minimizes the cost function of the model. The most well-known loss functions, like MSE or ...
0
votes
2answers
602 views

Gradient Descent Python Implementation isnt converging

I trying to implement gradient descent in Python and I am following andrew ng course in order to follow the math. However, my implementation isnt working as I expect it to. It would be great you the ...
1
vote
1answer
92 views

How to Define a Cost Fucntion?

I want to define a cost function in python to identify optimum value in days when i should end a marketing campaign to save spend on campaigns not generating traffic good traffic. Problem is I dont ...
1
vote
0answers
14 views

How to optimize the separation of two distributions from binary classfication

Given a sample where for each individual a classification is predetermined (e.g. sick or not) and 5 random variables are measured. The random variables are on the same scale but from differnt bins. E....
3
votes
4answers
481 views

Is gradient descent slower for finite differences?

In gradient descent, we updated each parameter $\theta_i$ in the direction which minimizes a function $f(\theta_1,\theta_2,\dots,\theta_N)$ by doing $$\theta_1 \leftarrow \theta_1 - \alpha \frac{\...
4
votes
2answers
110 views

Why Gradient methods work in finding the parameters in Neural Networks?

After reading quite a lot of papers (20-30 or so), I feel that I am quite not understanding things. Let us focus on the supervised learnings (for example). Given a set of data $\mathcal{D}_{train}=\{...
1
vote
1answer
219 views

Why perceptron does not converge on data not linearly separable

This is how I understand the perceptron algorithm. The perceptron loss function is the hinge loss $\ell(w,x,y) = \max(0, -yw\cdot x)$. Suppose the data set is $D = \{(x_1,y_1),\dots,(x_n,y_n)\}$ with ...
1
vote
2answers
902 views

Difference between Gradient Descent and Normal Equation in Linear Regression

Hi I am new to Linear Regression. I want to know what is the difference b/w Gradient Descent and Mean Square Error in Linear Regression using machine learning? And When to use Gradient ...
5
votes
1answer
3k views

clipping the reward for adam optimizer in keras

I would like to clip the reward in keras. I saw it is possible to clip the norm and clip the value is sgd as follows: ...
1
vote
1answer
104 views

What is the significance of underflow during parameter update using stochastic gradient descent?

Background I am using scikit-learn's MLPRegressor to learn a model with the following arguments: ...
2
votes
1answer
616 views

multilayer perceptron do not converge

I have been coding my own multi layer perceptron in MATLAB and it can be compiled without error. My training data features,x, has values from 1 to 360, and training data output, y, has the value of ...
2
votes
1answer
89 views

Backpropagation

I use chain rule when doing backpropagation and then I do Gradient Descent with weighting coefficient and I am updating the weight, so I do not understand how the method works in the equations below. ...
7
votes
1answer
12k views

What is the difference between SGD classifier and the Logisitc regression?

To my understanding, the SGD classifier, and Logistic regression seems similar. An SGD classifier with loss = 'log' implements Logistic regression and loss = 'hinge' implements Linear SVM. I also ...
1
vote
0answers
103 views

RMSProp Optimizer Performing Poorly

I am building an RNN and have decided to try RMSProp as an alternative to sgd. Here is my implementation: ...
2
votes
1answer
70 views

Why are optimization algorithms slower at critical points?

I just found the animation below from Alec Radford's presentation: As visible, all algorithms are considerably slowed down at saddle point (where derivative is 0) and quicken up once they get out of ...
2
votes
0answers
166 views

Explanations about ADAM Optimizer algorithm

I'm a beginner in Machine learning and i'm searching for some optimizer for the gradient descent. I've searched many topics about that, and did a state of art of all these optimizers. I have just one ...
0
votes
1answer
47 views

Differentiating roadmap of a loss function

Let's say I'm performing Stochastic Gradient Descent (SGD) on binary cross entropy error while optimizing weight $w_{2}$. Binary cross entropy error: $$L(y|p(x_{i}))=-y_{i}*ln(p(x_{i}))-(1-y_{i})*...
2
votes
1answer
88 views

Gradient descent multidimensional linear regression - does learning rate affects concurrency? [closed]

I wonder if gradient descent for multidimensional regression always finds the right result? I feel like this doesn't always have to be true. I have done some calculations and actually got correct ...
39
votes
5answers
45k views

What is the difference between Gradient Descent and Stochastic Gradient Descent?

What is the difference between Gradient Descent and Stochastic Gradient Descent? I am not very familiar with these, can you describe the difference with a short example?
0
votes
1answer
215 views

Algorithm for backpropagation through time

I am reading through this article trying to understand the bptt algorithm, in the context of an RNN. However there is one part I don’t understand: ...
0
votes
1answer
29 views

What is the best statistical measure tool to measure how close data is to fitted regression line if outliers are not fitted

I am using a custom algorithm based on Gradient descent which computes the best fit on a training dataset. In this data set I have outliers i.e. data points that I do not want to fit. The algorithm is ...
1
vote
2answers
123 views

What is the intuition behind Ridge Regression and Adapting Gradient Descent algorithms?

So I was going through Adaptive Gradient Descent, and learning the intuition behind it: optimizing the learning algorithm, and getting the model to converge faster. The way AdaGrad does this, is by ...
1
vote
2answers
3k views

How the combination of cross entropy loss and gradient descent penalizes and rewards

For a simple problem of classification (C classes) using the softmax classifier, most people use the cross-entropy loss function to quantify the objective. The cross-entropy loss is: $$L = -\sum_{i=1}...

1 2 3
4
5
7