Questions tagged [gradient-descent]

Gradient Descent is an algorithm for finding the minimum of a function. It iteratively calculates partial derivatives (gradients) of the function and descends in steps proportional to those partial derivatives. One major application of Gradient Descent is fitting a parameterized model to a set of data: the function to be minimized is an error function for the model.

Filter by
Sorted by
Tagged with
3
votes
1answer
51 views

Will stochastic gradient descent converge for multivariate linear regression

I am trying to figure out if stochastic gradient descent for a multivariate linear regression will converge (assuming there is no mini-batching, i.e., the batch size is 1). My guess is yes, based on ...
3
votes
1answer
52 views

Why Gaussian mixture model uses Expectation maximization instead of Gradient descent?

Why Gaussian mixture model uses Expectation maximization instead of Gradient descent? What other models uses Expectation maximization to find best optimal parameters instead of using gradient descent?
0
votes
0answers
6 views

what are Latent model and Latent Varibles?

what are latent models and why they use Expectation maximization instead of gradient descent to optmize the parameters ? How to interpret and understand whether a hidden varible is present in model or ...
2
votes
1answer
106 views

How similar is Adam optimization and Gradient clipping?

According to the Adam optimization update rule: $$m \leftarrow \beta_1 m + (1 - \beta_1)\nabla J(\theta)$$ $$v \leftarrow \beta_2 v + (1 - \beta_2)(\nabla J(\theta) \odot \nabla J(\theta))$$ $$\theta \...
3
votes
2answers
236 views

Difference between RMSProp and Momentum?

Can someone please tell me the clear difference between the approaches of RMSProp and Gradient Descent with Momentum ? Both try to achieve the same effect . One of the blogs that I read states the ...
2
votes
1answer
44 views

XGBoost is it possible to prevent a feature from being used twice in the same tree?

I'm using XGBoost and all its doing is using the feature in the first column of my data. My feature importance chart correlates perfectly to the position of the feature in my xtrain. If I shuffle the ...
0
votes
0answers
41 views

clarification on back-propagation calculations for a fully connected neural network

I am currently taking Andrew Ng's Deep Learning Course on coursera and I couldn't get my head around how actually back-propagation in calculated. Let's say my fully connected neural network looks like ...
0
votes
0answers
18 views

Vectorised Neural Network Backprapogation

I've started the CS231n course and have been going through the proofs and derivations for backpropagation. These two papers by the lecturers are what I've been reading: Paper 1: - http://cs231n....
0
votes
1answer
27 views

Neural Network not learning when more than 1 training data is given

I am very new to neural networks and data science in general and wanted to try getting my hand in making a simple neural network in python. I tried to make a neural network from scratch hoping to ...
0
votes
0answers
20 views

When scaled conjugate gradient backpropagation is suitable for backpropagation?

I usually use gradient descent with Adam optimizer to perform backpropagation in deep learning methods. I knew it is a very efficient method. The question is in which situations we can use "scaled ...
1
vote
1answer
57 views

Why does Siamese neural networks use tied weights and how do they work?

Reading this paper on one-shot learning "Siamese Neural Networks for One-shot Image Recognition" I was introduced to the idea of Siamese Neural Networks. What I did not fully grasp was what they ...
1
vote
1answer
21 views

Has Gradient decent any effects on the actual predictions or designed for practical convenience?

If we consider Batch Gradient, Stochastic Gradient, Mini-batch Gradient, will they affect on the actual predictions? as we know they always attempt to reach a local minimum.
1
vote
0answers
56 views

Derivative of Loss wrt bias term

I read this and have an ambiguity. I try to understand well how to calculate the derivative of Loss w.r.t to bias. In this question, we have this definition: ...
0
votes
1answer
48 views

Neural Net Backprop Weight updating Pseudo code help please

Here is my code for Backpropagation weight updating. It's a simple network with 1 hidden layer and 1 output neuron. The activation function of both hidden and output layer uses tanh. I propagate the ...
1
vote
0answers
18 views

Tuning parameters for gradient boosting/xgboost

In practice, which parameter do you typically tune first? Do you tune the learning rate (or step size) first? and then tune the total number of iterations? And how do you go about tuning these ...
0
votes
1answer
19 views

how does gradient descent update weigths in neural network

Im currently trying to learn about back propagation, and it's going forward, but theres one thing that keeps me scratching my head, and doesnt really seems to be answered in any of the videos or ...
0
votes
0answers
31 views

How to compute error (instead of gradient of error) for each node in backpropagation?

I was going through the relevant chain rule mathematics and I have successfully implemented backpropagation from scratch for MNIST (once, I even tried doing this for a small sample data I created by ...
0
votes
0answers
16 views

Averaging biased gradient information?

Consider the following scenario: I want to estimate the gradient at a point $P$ in $\mathbb{R}^2$, and I have access to two pieces of directional information, the vectors $u$ and $v$. When we can ...
0
votes
1answer
58 views

Gradient of a function in Python

I've defined a function in this way: def qfun(par): return(par[0]+atan(par[3])*par[1]+atan(par[4])*par[2]) How can I obtain the gradient of this function ...
1
vote
1answer
68 views

Why is my loss increasing in gradient descent?

When the learning rate is 0.01 the loss seems to be decreasing whereas when I increase the learning rate even slightly, the loss increases. Why does this happen? Are the gradients calculated wrong? ...
0
votes
0answers
41 views

'loss: nan' during training of Neural Network in python

I am training a neural network in python. But the accuracy is very low. ...
1
vote
1answer
88 views

What is the gradient descent rule using binary cross entropy (BCE) with tanh?

Similar to this post, I need the gradient descent step of tanh but now with binary cross entropy (BCE). So we have $$ \Delta \omega = -\eta \frac{\delta E}{\delta \omega} $$ Now we have BCE: $$ ...
0
votes
0answers
14 views

Convergence check of constrained biconvex optimization problem

I participate in the development of a matrix factorization algorithm and I have some convergence issues. Here is the kind of minimization problem I am facing : $L(D,A) = ||X-DA||^2 + \sum_{(i,j) \in ...
0
votes
1answer
175 views

Mini Batch Gradient Descent shuffling

My data set is of shape (60,784,1000) with mini batches for input and (60,10,1000) for labels, should I shuffle only the 60 mini batches or the training examples themselves?
1
vote
1answer
99 views

Reason for capping Learning Rate (alpha) up to 1 for Gradient Descent

I am learning to implement Gradient Descent algorithm in Python and came across the problem of selecting the right learning rate. I have learned that learning rates are usually selected up to 1 (...
0
votes
0answers
26 views

Basic questions about fitting a formula with gradient descent or genetic algorythm

I've been trying to code a following problem. I have defined a function depending on a number of parameters (in my case, those of a Bragg mirror and a x-ray beam). Now I am trying to compare the ...
1
vote
1answer
25 views

Displaying network error as a single value

I've been writing a neural network from scratch. I've completed the feedforward, backpropagation, and mini-batch gradient descent methods, so I can train the network. Other neural networks I've worked ...
0
votes
1answer
19 views

How can we get gradient with some other loss function apart from MSE?

In most of the gradient search examples, the update to weights are done by subtracting the derivative of MSE. Can we have an example, where we did not use ...
1
vote
0answers
15 views

Matlab Optimization. Meaning of warning: “The slope should be 2. It appears to be 1.”

I'm using the manopt package to solve some optimization problems in matlab. The problem is of the form. problem.cost = @(x) f(x) problem.egrad = @(x) g(x) After the problem definition, I check ...
2
votes
0answers
45 views

Dissecting and understanding the Adam optimization's formula

Adam's optimization has the fololwing parameter update rule : $$ \theta_{t+1} = \theta_{t} - \alpha*\dfrac{m_t}{\sqrt{v_t + \epsilon}}$$ where $$ m_t \text{ is first moment of gradients and} \space ...
3
votes
1answer
45 views

Gradient Checking: MeanSquareError. Why huge epsilon improves discrepancy?

I am using custom C++ code, and coded a simple "Mean Squared Error" layer. Temporarily using it for the 'classification task', not a simple regression. ...maybe this causes the issues? I don't have ...
0
votes
0answers
37 views

SGDClassifier on Big Data

I am trying actually to train a SGDClassifier with over 4,000,000 samples of data without any positive results. X vector has 6 features and looks like : [ 2 , 4 , 56431555 , 1 , 0 , 33] Y vector has ...
0
votes
0answers
76 views

Regression by PhasedLSTM with a gradient explosion

I found PhasedLSTM inspirational, and used it (PLSTM: Phased LSTM in Keras) to perform the regression (to find the correlation between an input sequence and an output sequence), with Adam optimizer <...
0
votes
0answers
13 views

huge disbalance between Dc and Dm in gradient descent

i get a huge gap in my starting position between Dc and Dm and although they both go down in each itteration,still the gap is so huge that when one comes close to zero the other is still very large ...
0
votes
0answers
18 views

gradient descent regression in matrices form diverges

I am trying to build fit the best fit for my random distribution. I have done exactly by the formulas in the book shown bellow. I get divergence in the error function. Where did I go wrong? my Matlab ...
0
votes
0answers
24 views

Neural network not passing gradient check

I'm following a course by Andrew ng in coursera, and while doing the course I'm also trying to implement a more sophisticated neural network along the way, so I can use what I've been learning, my ...
3
votes
1answer
377 views

How to prevent vanishing gradient or exploding gradient?

Whats causing the vanishing gradient or exploding gradient to occur, and what are the measures to be taken to prevent it?
1
vote
0answers
15 views

Gradient calculation for proportional odds (logistic) model

I am trying to calculate a gradient for a proportional odds model. http://fa.bianp.net/blog/2013/logistic-ordinal-regression/ What steps are required to take the derivative with respect to w? $$\...
0
votes
1answer
30 views

Finding a vector that minimize the MSE of its linear combination

I have been doing a COVID-19 related project. Here is the question: N = vector of daily new infected cases D = vector of daily deaths E[D] = estimation of daily deaths N is a n-dimensional vector, n ...
1
vote
1answer
68 views

gradient descent diverges extremely

I have manually created a random data set around some mean value and I have tried to use gradient descent linear regression to predict this simple mean value. I have done exactly like in the manual ...
0
votes
2answers
70 views

gradient descent in n dimensions

Gradient descent in $n$ dimensions. I'm learning about the downward gradient and the youtube videos and books only show a 2d curve as the slope drops to the minimum of the curve. My question is, ...
1
vote
0answers
50 views

Transposed Convolution without using Python built-in functions

Amateur here: How can we write a 2D transposed convolution (aka deconvolution) using the steepest descent method given the following restrictions: cannot use any Python built-in functions cannot ...
0
votes
1answer
31 views

Gradient for starting Backpropagation

I was reading this nice tutorial about Pytorch's basics: https://pytorch.org/tutorials/beginner/pytorch_with_examples.html In the first example (pure Numpy), the author starts the backward phase by ...
1
vote
0answers
188 views

grad-cam implementation on mobilenet SSD network

Below is a gradcam implementation for a standard image classifier : ...
0
votes
0answers
29 views

Does severe multicollinearity affect solving linear regression by gradient descent?

Since OLS may fail when there is severe/near perfect multicollinearity, how would gradient descent perform in such a scenario? Does it converge at the minima? (My guess is, Cost function of linear ...
0
votes
0answers
53 views

Tensorflow - Manually decay Adam optimizer

I've been experimenting with reinforcement learning and using the train_on_batch method of tf.keras.models.Model to update the ...
0
votes
0answers
40 views

Projected gradient descent in keras

I am currently working on a project and I need to do project gradient descent instead of vanilla gradient descent on a network. I am unsure if current deep learning frameworks have that functionality. ...
4
votes
2answers
231 views

What is the best way to find minima in Logistic regression?

In the Andrew NG's tutorial on Machine Learning, he takes the first derivative of the error function then takes small steps in direction of the derivative to find minima. (Gradient Descent basically) ...
0
votes
1answer
41 views

Should bias updates be porportional to overfitting?

According to questions on the internet, the bias is a learnable parameter, and there are different solutions to updating it, but I failed to find a concise methodology of correctly updating biases ...
1
vote
1answer
24 views

line search in gradient descent applied to a convex function

I have been working on implementing a line search method for gradient descent where I made an assumption that at any given point on my surface of the loss function I can reach the minima by the single ...

1
2
3 4 5
8