Questions tagged [gradient-descent]

Gradient Descent is an algorithm for finding the minimum of a function. It iteratively calculates partial derivatives (gradients) of the function and descends in steps proportional to those partial derivatives. One major application of Gradient Descent is fitting a parameterized model to a set of data: the function to be minimized is an error function for the model.

Filter by
Sorted by
Tagged with
2
votes
0answers
33 views

Learning parameters when loss is a piecewise function

I have a network to generate a single number $T$. I know in advance: a property of the loss function is that, when $T \in [a_1, a_2]$, the loss has the same value $L_1$; when $T \in [a_2, a_3]$, the ...
0
votes
0answers
30 views

Why L2 norm in AdaGrad update equation not L1?

The update equation of AdaGrad is as follows: I understand that sparse features have small updates and this is a problem. I understand that the idea of AdaGrad is to make the update speed (learning ...
0
votes
1answer
13 views

Is it valid to use numpy.gradient to find slope of line as well as slope of curve at any point?

what is the difference between slope of line and slope of curve ? Is it valid to use numpy.gradient to find slope of line and slope of curve at any point ? ...
1
vote
1answer
20 views

How to find slope of curve at certain points

how to find slope at certain points circled in blue in below curve ? Are these below 2 approaches valid ? though they give different results . How to automatically find the points where the slope ...
2
votes
1answer
26 views

Will stochastic gradient descent converge for multivariate linear regression

I am trying to figure out if stochastic gradient descent for a multivariate linear regression will converge (assuming there is no mini-batching, i.e., the batch size is 1). My guess is yes, based on ...
1
vote
1answer
32 views

Why Gaussian mixture model uses Expectation maximization instead of Gradient descent?

Why Gaussian mixture model uses Expectation maximization instead of Gradient descent? What other models uses Expectation maximization to find best optimal parameters instead of using gradient descent?
0
votes
0answers
6 views

what are Latent model and Latent Varibles?

what are latent models and why they use Expectation maximization instead of gradient descent to optmize the parameters ? How to interpret and understand whether a hidden varible is present in model or ...
1
vote
1answer
24 views

How similar is Adam optimization and Gradient clipping?

According to the Adam optimization update rule: $$m \leftarrow \beta_1 m + (1 - \beta_1)\nabla J(\theta)$$ $$v \leftarrow \beta_2 v + (1 - \beta_2)(\nabla J(\theta) \odot \nabla J(\theta))$$ $$\theta \...
3
votes
2answers
73 views

Difference between RMSProp and Momentum?

Can someone please tell me the clear difference between the approaches of RMSProp and Gradient Descent with Momentum ? Both try to achieve the same effect . One of the blogs that I read states the ...
2
votes
1answer
18 views

XGBoost is it possible to prevent a feature from being used twice in the same tree?

I'm using XGBoost and all its doing is using the feature in the first column of my data. My feature importance chart correlates perfectly to the position of the feature in my xtrain. If I shuffle the ...
0
votes
0answers
36 views

clarification on back-propagation calculations for a fully connected neural network

I am currently taking Andrew Ng's Deep Learning Course on coursera and I couldn't get my head around how actually back-propagation in calculated. Let's say my fully connected neural network looks like ...
0
votes
0answers
17 views

Vectorised Neural Network Backprapogation

I've started the CS231n course and have been going through the proofs and derivations for backpropagation. These two papers by the lecturers are what I've been reading: Paper 1: - http://cs231n....
0
votes
1answer
23 views

Neural Network not learning when more than 1 training data is given

I am very new to neural networks and data science in general and wanted to try getting my hand in making a simple neural network in python. I tried to make a neural network from scratch hoping to ...
0
votes
0answers
10 views

When scaled conjugate gradient backpropagation is suitable for backpropagation?

I usually use gradient descent with Adam optimizer to perform backpropagation in deep learning methods. I knew it is a very efficient method. The question is in which situations we can use "scaled ...
0
votes
1answer
21 views

Why does Siamese neural networks use tied weights and how do they work?

Reading this paper on one-shot learning "Siamese Neural Networks for One-shot Image Recognition" I was introduced to the idea of Siamese Neural Networks. What I did not fully grasp was what they ...
1
vote
1answer
20 views

Has Gradient decent any effects on the actual predictions or designed for practical convenience?

If we consider Batch Gradient, Stochastic Gradient, Mini-batch Gradient, will they affect on the actual predictions? as we know they always attempt to reach a local minimum.
0
votes
1answer
29 views

Neural Net Backprop Weight updating Pseudo code help please

Here is my code for Backpropagation weight updating. It's a simple network with 1 hidden layer and 1 output neuron. The activation function of both hidden and output layer uses tanh. I propagate the ...
1
vote
0answers
16 views

Tuning parameters for gradient boosting/xgboost

In practice, which parameter do you typically tune first? Do you tune the learning rate (or step size) first? and then tune the total number of iterations? And how do you go about tuning these ...
0
votes
1answer
19 views

how does gradient descent update weigths in neural network

Im currently trying to learn about back propagation, and it's going forward, but theres one thing that keeps me scratching my head, and doesnt really seems to be answered in any of the videos or ...
0
votes
0answers
30 views

How to compute error (instead of gradient of error) for each node in backpropagation?

I was going through the relevant chain rule mathematics and I have successfully implemented backpropagation from scratch for MNIST (once, I even tried doing this for a small sample data I created by ...
0
votes
0answers
13 views

Averaging biased gradient information?

Consider the following scenario: I want to estimate the gradient at a point $P$ in $\mathbb{R}^2$, and I have access to two pieces of directional information, the vectors $u$ and $v$. When we can ...
0
votes
1answer
30 views

Gradient of a function in Python

I've defined a function in this way: def qfun(par): return(par[0]+atan(par[3])*par[1]+atan(par[4])*par[2]) How can I obtain the gradient of this function ...
1
vote
1answer
25 views

Why is my loss increasing in gradient descent?

When the learning rate is 0.01 the loss seems to be decreasing whereas when I increase the learning rate even slightly, the loss increases. Why does this happen? Are the gradients calculated wrong? ...
0
votes
0answers
27 views

'loss: nan' during training of Neural Network in python

I am training a neural network in python. But the accuracy is very low. ...
1
vote
1answer
18 views

What is the gradient descent rule using binary cross entropy (BCE) with tanh?

Similar to this post, I need the gradient descent step of tanh but now with binary cross entropy (BCE). So we have $$ \Delta \omega = -\eta \frac{\delta E}{\delta \omega} $$ Now we have BCE: $$ ...
0
votes
0answers
13 views

Convergence check of constrained biconvex optimization problem

I participate in the development of a matrix factorization algorithm and I have some convergence issues. Here is the kind of minimization problem I am facing : $L(D,A) = ||X-DA||^2 + \sum_{(i,j) \in ...
0
votes
1answer
48 views

Mini Batch Gradient Descent shuffling

My data set is of shape (60,784,1000) with mini batches for input and (60,10,1000) for labels, should I shuffle only the 60 mini batches or the training examples themselves?
1
vote
1answer
51 views

Reason for capping Learning Rate (alpha) up to 1 for Gradient Descent

I am learning to implement Gradient Descent algorithm in Python and came across the problem of selecting the right learning rate. I have learned that learning rates are usually selected up to 1 (...
0
votes
0answers
26 views

Basic questions about fitting a formula with gradient descent or genetic algorythm

I've been trying to code a following problem. I have defined a function depending on a number of parameters (in my case, those of a Bragg mirror and a x-ray beam). Now I am trying to compare the ...
1
vote
1answer
24 views

Displaying network error as a single value

I've been writing a neural network from scratch. I've completed the feedforward, backpropagation, and mini-batch gradient descent methods, so I can train the network. Other neural networks I've worked ...
0
votes
0answers
9 views

How can we get gradient with some other loss function apart from MSE?

In most of the gradient search examples, the update to weights are done by subtracting the derivative of MSE. Can we have an example, where we did not use derivative of MSE, but used derivative of ...
0
votes
0answers
10 views

Matlab Optimization. Meaning of warning: “The slope should be 2. It appears to be 1.”

I'm using the manopt package to solve some optimization problems in matlab. The problem is of the form. problem.cost = @(x) f(x) problem.egrad = @(x) g(x) After the problem definition, I check ...
2
votes
0answers
30 views

Dissecting and understanding the Adam optimization's formula

Adam's optimization has the fololwing parameter update rule : $$ \theta_{t+1} = \theta_{t} - \alpha*\dfrac{m_t}{\sqrt{v_t + \epsilon}}$$ where $$ m_t \text{ is first moment of gradients and} \space ...
0
votes
0answers
21 views

SGDClassifier on Big Data

I am trying actually to train a SGDClassifier with over 4,000,000 samples of data without any positive results. X vector has 6 features and looks like : [ 2 , 4 , 56431555 , 1 , 0 , 33] Y vector has ...
0
votes
0answers
39 views

Regression by PhasedLSTM with a gradient explosion

I found PhasedLSTM inspirational, and used it (PLSTM: Phased LSTM in Keras) to perform the regression (to find the correlation between an input sequence and an output sequence), with Adam optimizer <...
0
votes
0answers
10 views

huge disbalance between Dc and Dm in gradient descent

i get a huge gap in my starting position between Dc and Dm and although they both go down in each itteration,still the gap is so huge that when one comes close to zero the other is still very large ...
0
votes
0answers
18 views

gradient descent regression in matrices form diverges

I am trying to build fit the best fit for my random distribution. I have done exactly by the formulas in the book shown bellow. I get divergence in the error function. Where did I go wrong? my Matlab ...
0
votes
0answers
21 views

Neural network not passing gradient check

I'm following a course by Andrew ng in coursera, and while doing the course I'm also trying to implement a more sophisticated neural network along the way, so I can use what I've been learning, my ...
3
votes
1answer
79 views

How to prevent vanishing gradient or exploding gradient?

Whats causing the vanishing gradient or exploding gradient to occur, and what are the measures to be taken to prevent it?
1
vote
0answers
14 views

Gradient calculation for proportional odds (logistic) model

I am trying to calculate a gradient for a proportional odds model. http://fa.bianp.net/blog/2013/logistic-ordinal-regression/ What steps are required to take the derivative with respect to w? $$\...
0
votes
1answer
24 views

Finding a vector that minimize the MSE of its linear combination

I have been doing a COVID-19 related project. Here is the question: N = vector of daily new infected cases D = vector of daily deaths E[D] = estimation of daily deaths N is a n-dimensional vector, n ...
0
votes
1answer
61 views

gradient descent diverges extremely

I have manually created a random data set around some mean value and I have tried to use gradient descent linear regression to predict this simple mean value. I have done exactly like in the manual ...
0
votes
2answers
43 views

gradient descent in n dimensions

Gradient descent in $n$ dimensions. I'm learning about the downward gradient and the youtube videos and books only show a 2d curve as the slope drops to the minimum of the curve. My question is, ...
1
vote
0answers
32 views

Transposed Convolution without using Python built-in functions

Amateur here: How can we write a 2D transposed convolution (aka deconvolution) using the steepest descent method given the following restrictions: cannot use any Python built-in functions cannot ...
0
votes
1answer
20 views

Gradient for starting Backpropagation

I was reading this nice tutorial about Pytorch's basics: https://pytorch.org/tutorials/beginner/pytorch_with_examples.html In the first example (pure Numpy), the author starts the backward phase by ...
1
vote
0answers
91 views

grad-cam implementation on mobilenet SSD network

Below is a gradcam implementation for a standard image classifier : ...
0
votes
0answers
11 views

Does severe multicollinearity affect solving linear regression by gradient descent?

Since OLS may fail when there is severe/near perfect multicollinearity, how would gradient descent perform in such a scenario? Does it converge at the minima? (My guess is, Cost function of linear ...
0
votes
0answers
25 views

Tensorflow - Manually decay Adam optimizer

I've been experimenting with reinforcement learning and using the train_on_batch method of tf.keras.models.Model to update the ...
0
votes
0answers
16 views

Projected gradient descent in keras

I am currently working on a project and I need to do project gradient descent instead of vanilla gradient descent on a network. I am unsure if current deep learning frameworks have that functionality. ...
3
votes
2answers
221 views

What is the best way to find minima in Logistic regression?

In the Andrew NG's tutorial on Machine Learning, he takes the first derivative of the error function then takes small steps in direction of the derivative to find minima. (Gradient Descent basically) ...

1
2 3 4 5
7