Questions tagged [gradient-descent]

Gradient Descent is an algorithm for finding the minimum of a function. It iteratively calculates partial derivatives (gradients) of the function and descends in steps proportional to those partial derivatives. One major application of Gradient Descent is fitting a parameterized model to a set of data: the function to be minimized is an error function for the model.

Filter by
Sorted by
Tagged with
2
votes
0answers
23 views

Linear Regression in Python using gradient descent

I am trying to implement a simple multivariate linear regression model without using any inbuilt machine libraries. So far, I have been able to get a root mean squared error for training about $2.93$ ...
0
votes
1answer
24 views

Gradient Descent or Normal Equation?

Hi guys I am really struggling with this question. I need to pick the correct choice: Suppose you have a dataset with m = 50 examples and n = 15 features for each example. You want to use ...
0
votes
2answers
55 views

What's the proper way to do back propagation in Deep Fully Connected Neural Network for binary classification

I tried to implement a Deep fully connected neural network for binary classification using python and numpy and used Gradient Descent as optimization algorithm. ...
1
vote
3answers
52 views

Gradient descent in a noisy environment

How to know the right direction in a noisy environment? In the typical example of neural network learning, we can see several local minima. The gradient descent is choosing one local minimum and ...
0
votes
0answers
16 views

problem with vanishing/exploding gradient problems solution

I have few doubts around vanishing/exploding gradients. The problem with vanishing gradient is, When the weights are randomly initialized in a deep network, During back propagation initial layers ...
0
votes
0answers
6 views

independent training algorithms for neural network in Matlab

The picture below shows some of training algorithms in Matlab used to train a neural network. These algorithms can be classified into 3 main families: Gradient Descent, Conjugate Gradient Descent and ...
-1
votes
0answers
31 views

Want to apply 5-fold cross validation on a dataset without using sklearn in Python

I have a dataset with 4177 entries in CSV format. I imported the file using pandas in python. I want to apply a 5-fold validation on it and compare the Root mean ...
2
votes
1answer
27 views

SGD, calculating it by hand

While I find a lot of material of SGD (Stochastic gradient descent), I am struggling to find one concrete example with numbers e.g. calculating it by hand for let's say, one iteration would help me a ...
2
votes
1answer
105 views

Gradient descent formula

I came across an interesting book about neural network basics, and the formula for gradient descent from one of the first chapters says: Gradient descent: For each layer update the weights ...
0
votes
0answers
7 views

Backtracking Line search for Multiclass classification gradient descent

For my case i am dealing with multiclass problem and there are total 28 direction component for each class and there are total 5 classes, for given equation above, f(w+nd) and f(w) gives scaler values ...
3
votes
1answer
53 views

Fast Python implementation of the gradient descent

I'm looking for fast Python implémentations of gradient descent optimization algorithm. I have a convex problem , with no constraint, so for now I'm using the BFGS algorithm implemented in scikit-...
1
vote
0answers
73 views

SKLearn Boston dataset gradient descent not working

I am trying to compare some simple methods for linear regression as an exercise. I have already used LinearRegression from the SKLearn library in python as well as the formula of linear regression. ...
2
votes
1answer
17 views

Quadratic approximation of L1 regularized cost function

I'm reading the Deep Learning book of Goodfellow, but I fail to see why minimization of (7.22) gives (7.23). I tried to compute the gradient w.r.t. the $w_{i}$ and set this to zero, but it doesn't ...
4
votes
2answers
40 views

Question of using gradient descent instead of calculus. I checked previous questions there are still points to clarify

First of all I checked http://stats.stackexchange.com/questions/23128/solving-for-regression-parameters-in-closed-form-vs-gradient-descent, http://stackoverflow.com/questions/26804656/why-do-we-use-...
5
votes
2answers
67 views

Why is taking the gradient of the average error in SGD not correct, but rather the average of the gradients of single errors?

I am a little confused about taking averages in cost functions and SGD. So far I always thought in SGD you would compute the average error for a batch and then backpropagate it. But then I was told in ...
0
votes
0answers
14 views

Does a max-pooling layer in a ConvNet contribute to the “vanishing gradient” problem?

I would answer no, but am not sure if I'm missing something and hope you can help me out: The derivative of a max-pooling layer in a ConvNet is one w.r.t. the maximum value and zero for all others. A ...
1
vote
0answers
32 views

Gradient boosting, where did the constant go?

In the very early papers on gradient boosting, the ensemble would include a constant and a sum of base learners i.e. $F(X) = a_0 + \sum\limits_{i} a_i f_i(X)$ The constant is fitted first (i.e. if ...
2
votes
2answers
54 views

Why does feature scaling improve the convergence speed for gradient descent?

From this article, it says: We can speed up gradient descent by scaling. This is because θ will descend quickly on small ranges and slowly on large ranges, and so will oscillate inefficiently down ...
0
votes
1answer
22 views

Derivative of activation function used in gradient descent algorithms

Why is it necessary to calculate the derivative of activation functions while updating model( regression or NN) parameters? Why is the constant gradient of linear functions considered as a ...
0
votes
1answer
21 views

Confused with the derivation of the gradient descent update rule

I have been going over some theory for gradient descent. The source I am looking at said that the change in cost can be described by the following equation: $$∆C=∇C∙∆w$$ where $∇C$ is the gradient ...
1
vote
1answer
17 views

Pytorch - Gradient distribution between functions

https://colab.research.google.com/github/pytorch/tutorials/blob/gh-pages/_downloads/neural_networks_tutorial.ipynb Hi I am trying to understand the NN with pytorch. I have doubts in gradient ...
1
vote
0answers
9 views

Why does NAG cause unstable validation loss?

I'm building a neural network for a classification problem. When playing around with some hyperparameters, I was surprised to see that using Nesterov's Accelerated Gradient instead of vanilla SGD ...
0
votes
1answer
17 views

What does this expression from gradient descent mean?

I am looking over some neural network theory and came across this equation, coupled with this description (gradient descent ball-valley analogy): ''let's think about what happens when we move the ...
1
vote
3answers
48 views

Different learning rates for each dimension

I have been thinking about why normalization and scaling are done for each feature in the basic context of gradient descent. One thing that got me wondering is that we use a pre-defined set of ...
0
votes
1answer
22 views

Does it make sense to train an Autoencoder for Dimensionality Reduction using Mini-Batch Gradient Descent?

I want to reduce the dimensionality of a dataset using a stacked Autoencoder. The size of the dataset and the computing power at my disposal make it very difficult to train the Network using simple, ...
1
vote
0answers
97 views

Updating Weight Using Updates on Related Data

Suppose $$ x=Ay $$ The $x$ is $M\times 1$, $y$ is $N \times 1$ and $A$ is $M\times N$ We have the data $x$ and would like to know what $y$ is. However, the matrix $A$ is too large for pseudo-...
0
votes
0answers
39 views

In an RNN, if the gradients don't vanish for long/distant terms, won't the derivative of the error be either divergent to infinity or oscillatory?

P.S. Crosss posted here- https://stats.stackexchange.com/questions/413843/in-an-rnn-if-the-gradients-dont-vanish-for-long-distant-terms-wont-the-deriv, as I've got no answer, I'm asking here: In my ...
0
votes
0answers
23 views

How to get the weights of a linear model by solving normal equation?

In chapter 6.1 of the book Deep Learning, the author tries to learn the XOR function by using a linear model (on page 168). Linear Model: $f(\mathbf{x};\mathbf{w},b)=\mathbf{x}^T\mathbf{w}+b$ MSE ...
2
votes
1answer
71 views

Dueling Network gradient with respect to Advantage stream

Looking at Dueling DQN: $Q = V + A - mean(A)$ For simplicity, let's assume we are working with 4 neurons. Recall that Value stream only has 1 neuron $(v_0)$ Re-writing the above equation, we get: $...
2
votes
2answers
113 views

Are mini batches sampled randomly in Keras' Sequential.fit method()

When you .fit a Keras Sequential() model, you can specify a batch_size parameter. I have ...
1
vote
3answers
181 views

How do GD, Batch GD, SGD, and Mini-Batch SGD differ?

How do these four types of gradient descent functions differ from each other? GD Batch GD SGD Mini-Batch SGD
2
votes
1answer
37 views

Intractability in Variational Autoencoders

I'm having difficulty understanding when integrals are intractable in variational inference problems. In a variational autoencoder with observation $x$ and latent variable $z$ we want to maximize ...
0
votes
0answers
20 views

Do we include the L2 regularization loss gradients when visualizing the norm of the gradients?

During training I need to plot the gradient norms at each layer to monitor the progress. When the loss function is made up of the main loss term plus the L2 regularization term, should we only plot ...
0
votes
0answers
26 views

Can one set manual adaptive learning in SGDRegressor()?

I wanted to update learning rate $r = r/2$ in each iteration of SGDRegressor(). I cannot find any way so far to update the learning rate manually. There is a choice called adaptive but it doesn't look ...
0
votes
1answer
30 views

Compute gradients in parallel

Here is part of my code: ...
0
votes
0answers
7 views

Calculate gradients in parallel

Here is part of my code: ...
0
votes
0answers
24 views

Can optimal intercept be gotten from optimal hyperplane in gradient descent?

I have read if $w_1^T$ is a plane and $b$ be intercept on $y$ axis. Then, $w_1^T.x_1+b=0$ if $w_1,x_1 \in R^{d}$ in $d$ dimensional hyperplane. Alternatively, $w^Tx =0$ if $w=<b, w_1>$ & $x=&...
2
votes
0answers
19 views

Get derivatives from your NN

How can I get the gradient of a node in the NN with respect to another one? I need to train a NN, which for the sake of simplicity has 2 neurons as input (x, y), a neuron as a bottleneck (z), and 2 ...
1
vote
1answer
24 views

How Stochastic Gradient Descent used like Mini Batch gradient descent?

As I know Gradient Descent has three variants which are: 1- Batch Gradient Descent: processes all the training examples for each iteration of gradient descent. 2- Stochastic Gradient Descent: ...
0
votes
0answers
19 views

How to create an autograd library from scratch like pytorch?

I am trying to implement a deep learning library from scratch. Most common DL framework uses autograd. Unfortunately, I haven't seen a lot of resources on how to create one autograd library. Is there ...
0
votes
0answers
30 views

Calculate Gradient in Machine Learning Classification Problem

I going through the optimization techniques in DL. I went through a code snippet of calculating the gradient. ...
1
vote
0answers
47 views

CartPole v1 - Simple backprop with 1 hidden layer

I'm trying to solve the CartPole-v1 problem from OpenAI by using backprop on a one-layer neural network - while updating the model at every time step using State action values (Q(s,a)). I'm unable to ...
1
vote
1answer
25 views

Why does my linear regression model converge to a non-zero gradient value?

I have a basic 2D Linear Regression model coded out (using gradient descent), yet it doesn't seem to work as well as it should. What I expect is that m and ...
1
vote
1answer
157 views

Am I using optim.SGD incorrectly in pytorch?

I am doing reinforcement learning in checkers. After each game the network beats itself, I calculate the loss of every individual position in the game, call backward(), and step(). I am beginning to ...
0
votes
0answers
16 views

What's the correct way of implementation of cost function and gradient function in logistical regression after regularisation?

This is the cost function of logistic regression: which i could implement correctly, with the code : ...
0
votes
0answers
19 views

Gradient of batchnorm layer

In recent paper about How Does Batch Normalization Help Optimization? by Satunkar et.al. The paper mentions facts about the derivative of a loss function through a batchnorm layer. The paper state ...
0
votes
0answers
20 views

my gradient decent isn't working right

i'm very new to machine learning, i was doing an exercise about classification using the sigmoid as hypothesis, i don't know what's wrong but my cost function keep increasing and the slope of the cost ...
4
votes
1answer
87 views

Gradient Descent in ReLU Neural Network

I’m new to machine learning and recently facing a problem on back propagation of training a neural network using ReLU activation function shown in the figure. My problem is to update the weights ...
1
vote
2answers
149 views

which metric is better for boosting methods

I work on a dataset of 300 000 samples and I try to make a comparison between logistic regression (with gradients descent) and a LightBoost for binary classification in order to choose the better one. ...
0
votes
0answers
17 views

How to retain dependency between variables in PyTorch?

I am modeling k-dimensional positions over time t = 0...T using a set of initial positions Z0 with requires_grad=True and storing the results in Z with requires_grad=False for the remaining T-1 time ...