Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [gradient-descent]

The tag has no usage guidance.

0
votes
1answer
11 views

What does the “randomly shuffle training samples” in stochastic gradient descent attain?

What does the "randomly shuffle training samples" in stochastic gradient descent attain? I interpreted that since the training samples are used to compute $$\hat{y}=f(w^t x)$$ so if the order of $x$...
0
votes
0answers
7 views

too few data while too many degrees of freedom in linear regression

To recognize handwritten digits, I have a fully connected network, containing only 2 layers: input layer (all pixels of the image) and output layer (0 or 1). I use the simplest linear regression for ...
0
votes
0answers
6 views

Newton's method gradient and hessian formulation difficulties

Logistic regression Newton's Method Newton Method Lecture II In this picture the logistic regression cost function , Newtons Method and gradient and Hessian is defined. How to get this function that ...
1
vote
1answer
27 views

What is non-decomposable and/or non-differentiable loss function?

I have been reading some deep learning literature and came up with these concepts of non-decomposable and non-differentiable loss functions. My question is are these same thing? if not how are they ...
0
votes
0answers
9 views

Per parameter learning rate for AdamOptimizer by scaling gradients

I'm using an AdamOptimizer, and I compute the gradients, but before applying the descent step, I scale (i.e.: multiply) gradients with constants, to mimic having a different learning rate per ...
0
votes
0answers
12 views

why don't SOTA neural network models back propagation use conjugate gradients?

My understanding is that the conjugate gradient method is faster than gradient descent because it does less zig zags while descending. How come the state of the art papers I see all use gradient ...
0
votes
1answer
54 views

Should the minimum value of a cost (loss) function be equal to zero?

We know optimization techniques search in the space of all the possible parameters for a parameter set that minimizes the cost function of the model. The most well-known loss functions, like MSE or ...
0
votes
1answer
38 views

Gradient Descent Python Implementation isnt converging

I trying to implement gradient descent in Python and I am following andrew ng course in order to follow the math. However, my implementation isnt working as I expect it to. It would be great you the ...
0
votes
1answer
33 views

How to Define a Cost Fucntion?

I want to define a cost function in python to identify optimum value in days when i should end a marketing campaign to save spend on campaigns not generating traffic good traffic. Problem is I dont ...
1
vote
0answers
12 views

How to optimize the separation of two distributions from binary classfication

Given a sample where for each individual a classification is predetermined (e.g. sick or not) and 5 random variables are measured. The random variables are on the same scale but from differnt bins. E....
2
votes
4answers
43 views

Is gradient descent slower for finite differences?

In gradient descent, we updated each parameter $\theta_i$ in the direction which minimizes a function $f(\theta_1,\theta_2,\dots,\theta_N)$ by doing $$\theta_1 \leftarrow \theta_1 - \alpha \frac{\...
4
votes
2answers
75 views

Why Gradient methods work in finding the parameters in Neural Networks?

After reading quite a lot of papers (20-30 or so), I feel that I am quite not understanding things. Let us focus on the supervised learnings (for example). Given a set of data $\mathcal{D}_{train}=\{...
0
votes
0answers
41 views

What are the reasons of select a optimizer to be SGD or Adam in DQN?Why?

I saw several comparison between SGD, RMSPROP and ADAM but what I am looking for is their comparsion in DQN algorithm? What is best to select as optimizer SGD or Adam in DQN? Why? Please check the ...
1
vote
0answers
37 views

Why perceptron does not converge on data not linearly separable

This is how I understand the perceptron algorithm. The perceptron loss function is the hinge loss $\ell(w,x,y) = \max(0, -yw\cdot x)$. Suppose the data set is $D = \{(x_1,y_1),\dots,(x_n,y_n)\}$ with ...
0
votes
1answer
98 views

Difference between Gradient Descent and Normal Equation in Linear Regression

Hi I am new to Linear Regression. I want to know what is the difference b/w Gradient Descent and Mean Square Error in Linear Regression using machine learning? And When to use Gradient ...
2
votes
0answers
31 views

clipping the reward for adam optimizer in keras

I would like to clip the reward in keras. I saw it is possible to clip the norm and clip the value is sgd as follows: ...
1
vote
0answers
16 views

What is the significance of underflow during parameter update using stochastic gradient descent?

Background I am using scikit-learn's MLPRegressor to learn a model with the following arguments: ...
1
vote
1answer
57 views

multilayer perceptron do not converge

I have been coding my own multi layer perceptron in MATLAB and it can be compiled without error. My training data features,x, has values from 1 to 360, and training data output, y, has the value of ...
2
votes
1answer
62 views

Backpropagation

I use chain rule when doing backpropagation and then I do Gradient Descent with weighting coefficient and I am updating the weight, so I do not understand how the method works in the equations below. ...
1
vote
1answer
502 views

What is the difference between SGD classifier and the Logisitc regression?

To my understanding, the SGD classifier, and Logistic regression seems similar. An SGD classifier with loss = 'log' implements Logistic regression and loss = 'hinge' implements Linear SVM. I also ...
1
vote
0answers
24 views

RMSProp Optimizer Performing Poorly

I am building an RNN and have decided to try RMSProp as an alternative to sgd. Here is my implementation: ...
2
votes
1answer
30 views

Why are optimization algorithms slower at critical points?

I just found the animation below from Alec Radford's presentation: As visible, all algorithms are considerably slowed down at saddle point (where derivative is 0) and quicken up once they get out of ...
0
votes
0answers
14 views

Why my Simple DL model with default SGD working very poorly in keras?

When I'm answering questions in StackOverflow, I found this question about the poor performance of DL model. I tried to implement the same code in my machine to see whats happening. I have also ...
3
votes
0answers
62 views

Explanations about ADAM Optimizer algorithm

I'm a beginner in Machine learning and i'm searching for some optimizer for the gradient descent. I've searched many topics about that, and did a state of art of all these optimizers. I have just one ...
0
votes
1answer
24 views

Differentiating roadmap of a loss function

Let's say I'm performing Stochastic Gradient Descent (SGD) on binary cross entropy error while optimizing weight $w_{2}$. Binary cross entropy error: $$L(y|p(x_{i}))=-y_{i}*ln(p(x_{i}))-(1-y_{i})*...
2
votes
1answer
35 views

Gradient descent multidimensional linear regression - does learning rate affects concurrency? [closed]

I wonder if gradient descent for multidimensional regression always finds the right result? I feel like this doesn't always have to be true. I have done some calculations and actually got correct ...
5
votes
3answers
2k views

What is the difference between Gradient Descent and Stochastic Gradient Descent?

What is the difference between Gradient Descent and Stochastic Gradient Descent? I do not know these things very well, can you describe it with a short example? Good luck with, Engin
0
votes
1answer
30 views

Algorithm for backpropagation through time

I am reading through this article trying to understand the bptt algorithm, in the context of an RNN. However there is one part I don’t understand: ...
0
votes
1answer
21 views

What is the best statistical measure tool to measure how close data is to fitted regression line if outliers are not fitted

I am using a custom algorithm based on Gradient descent which computes the best fit on a training dataset. In this data set I have outliers i.e. data points that I do not want to fit. The algorithm is ...
0
votes
0answers
145 views

Machine Learning: Stochastic gradient descent method for logistic regression in R

I am trying to write a code to solve the following problem (As stated in HW5 in the CalTech course Learning from Data): In this problem you will create your own target function f (probability in this ...
1
vote
1answer
35 views

What is the intuition behind Ridge Regression and Adapting Gradient Descent algorithms?

So I was going through Adaptive Gradient Descent, and learning the intuition behind it: optimizing the learning algorithm, and getting the model to converge faster. The way AdaGrad does this, is by ...
1
vote
2answers
468 views

How the combination of cross entropy loss and gradient descent penalizes and rewards

For a simple problem of classification (C classes) using the softmax classifier, most people use the cross-entropy loss function to quantify the objective. The cross-entropy loss is: $$L = -\sum_{i=1}...
1
vote
1answer
220 views

Linear Regression Optimization

I am learning linear regression right now. In the most of the examples of implementation of this method, which I found, gradient descent is used. Is there a better way to optimize linear regression ...
3
votes
1answer
41 views

Tensorflow Calculate error for a single neuron

I'm required to be able to calculate the error on a given neuron in a neural network using Tensorflow. Using this : ...
2
votes
1answer
28 views

Conflicting directions of weights gradients in gradient descent?

In a typical ANN backpropagation setting, we have multiple weights and we try to reduce the loss function by calculating the gradient of the function with respect to the weights let's say w1, w2, w3 ...
1
vote
1answer
30 views

Cost function dependence on size - batch gradient descent

I am applying the simple least mean square update rule using python but somehow the values of theta, I get, become very high. ...
2
votes
2answers
69 views

Why averaging the gradient works in Gradient Descent?

In Full-batch Gradient descent or Minibatch-GD we are getting gradient from several training examples. We then average them out to get a "high-quality" gradient, from several estimations and finally ...
1
vote
1answer
32 views

How to apply Gradient Descent to a summ of function

My target is to find a center of a circle that approximate a set of dots i want to find minimum of a function: $$\sum_{i=0}^N (\sqrt{(x_i - a)^2 + (y_i - b)^2} - R)^2$$ this function represent an ...
0
votes
0answers
12 views

Should the partials be averaged once or multiple times for backprop

I am trying to make a general purpose symbolic differentiation library, and I am not sure how to handle mini batch learning for SGD. When back-propagating, the sums of the partials are usually added ...
0
votes
0answers
25 views

Problem on the implementation of linear regression using tensorflow

I've implemented a basic linear regression with tensorflow and the bash gradient descent. It work perfectly but i wanted to see the regularization method on a over-fitted system. So I set a high-...
0
votes
0answers
119 views

How to understand the bias term b and how it is updated in ADALINE algorithm?

With $y=x*w+b$, where $x$ is the feature vector of a sample and $w$ is the weight vector, the update rule (SGD) for the bias $b$ is: $b \leftarrow b + \eta(o - y)$. With Gradient Descent, the final $...
3
votes
1answer
170 views

Using Mean Squared Error in Gradient Descent

I've recently been writing linear regression algorithms from scratch to gain an understanding of how the maths behind it works (something that was a bit of a black box beforehand), and so I got around ...
1
vote
2answers
34 views

Why do we double the number in a quadratic cost function or MSE?

$$ C(w,b) = \frac{1}{2n}\sum_{x}||y(x)-a||^2 $$ Where y is a 10-dimensional vector, a is the output, w is the weight and b is the bias and n is the number of inputs. If this is the MSE, shouldn't it ...
1
vote
0answers
11 views

Is there mathematical verification for TBPTT (truncated back propagation through time)

My question is in the title. I'm currently looking for a paper or academic reference to that algorithm, even in a novel framework. Thank you for all possible replies,
3
votes
1answer
158 views

Adam optimizer for projected gradient descent

The Adam optimizer is often used for training neural networks; it typically avoids the need for hyperparameter search over parameters like the learning rate, etc. The Adam optimizer is an improvement ...
2
votes
1answer
33 views

Do adaptive learning optimizers follow the steepest decent?

For adaptive learning optimizers such as Adam and RMSProb, the effective learning rate is not the same for all weight parameters. This means that we are not really following the direction of the ...
3
votes
1answer
438 views

Why do we use gradients instead of residuals in Gradient Boosting?

I have found mentions of two advantages in using gradients instead of actual residuals: 1) Using gradients will allow us to plug in any loss function (not just mse) without having to change our base ...
3
votes
1answer
116 views

Why Root Finding is important in Logistic Regression? (i.e. Newton Raphson)

I'd like to ask what is the main reason why we find the roots in logistic regression (i.e. why we use Newton Raphson method on logistic regression ). I understand the basics of Newton Raphson method, ...
2
votes
1answer
72 views

Why does Ensemble Averaging actually improve results?

Why does ensemble averaging work for neural networks? This is the main idea behind things like dropout. Consider an example of a hypersurface defined by the following image (white means lowest Cost). ...