Questions tagged [gradient-descent]

Gradient Descent is an algorithm for finding the minimum of a function. It iteratively calculates partial derivatives (gradients) of the function and descends in steps proportional to those partial derivatives. One major application of Gradient Descent is fitting a parameterized model to a set of data: the function to be minimized is an error function for the model.

Filter by
Sorted by
Tagged with
1
vote
1answer
16 views

Why does feature scaling improve the convergence speed for gradient descent?

From this article, it says: We can speed up gradient descent by scaling. This is because θ will descend quickly on small ranges and slowly on large ranges, and so will oscillate inefficiently down ...
0
votes
1answer
8 views

Derivative of activation function used in gradient descent algorithms

Why is it necessary to calculate the derivative of activation functions while updating model( regression or NN) parameters? Why is the constant gradient of linear functions considered as a ...
0
votes
1answer
13 views

Confused with the derivation of the gradient descent update rule

I have been going over some theory for gradient descent. The source I am looking at said that the change in cost can be described by the following equation: $$∆C=∇C∙∆w$$ where $∇C$ is the gradient ...
1
vote
1answer
11 views

Pytorch - Gradient distribution between functions

https://colab.research.google.com/github/pytorch/tutorials/blob/gh-pages/_downloads/neural_networks_tutorial.ipynb Hi I am trying to understand the NN with pytorch. I have doubts in gradient ...
1
vote
0answers
7 views

Why does NAG cause unstable validation loss?

I'm building a neural network for a classification problem. When playing around with some hyperparameters, I was surprised to see that using Nesterov's Accelerated Gradient instead of vanilla SGD ...
0
votes
1answer
16 views

What does this expression from gradient descent mean?

I am looking over some neural network theory and came across this equation, coupled with this description (gradient descent ball-valley analogy): ''let's think about what happens when we move the ...
1
vote
3answers
37 views

Different learning rates for each dimension

I have been thinking about why normalization and scaling are done for each feature in the basic context of gradient descent. One thing that got me wondering is that we use a pre-defined set of ...
0
votes
1answer
12 views

Does it make sense to train an Autoencoder for Dimensionality Reduction using Mini-Batch Gradient Descent?

I want to reduce the dimensionality of a dataset using a stacked Autoencoder. The size of the dataset and the computing power at my disposal make it very difficult to train the Network using simple, ...
1
vote
0answers
88 views

Updating Weight Using Updates on Related Data

Suppose $$ x=Ay $$ The $x$ is $M\times 1$, $y$ is $N \times 1$ and $A$ is $M\times N$ We have the data $x$ and would like to know what $y$ is. However, the matrix $A$ is too large for pseudo-...
0
votes
0answers
36 views

In an RNN, if the gradients don't vanish for long/distant terms, won't the derivative of the error be either divergent to infinity or oscillatory?

P.S. Crosss posted here- https://stats.stackexchange.com/questions/413843/in-an-rnn-if-the-gradients-dont-vanish-for-long-distant-terms-wont-the-deriv, as I've got no answer, I'm asking here: In my ...
0
votes
0answers
19 views

How to get the weights of a linear model by solving normal equation?

In chapter 6.1 of the book Deep Learning, the author tries to learn the XOR function by using a linear model (on page 168). Linear Model: $f(\mathbf{x};\mathbf{w},b)=\mathbf{x}^T\mathbf{w}+b$ MSE ...
2
votes
1answer
55 views

Dueling Network gradient with respect to Advantage stream

Looking at Dueling DQN: $Q = V + A - mean(A)$ For simplicity, let's assume we are working with 4 neurons. Recall that Value stream only has 1 neuron $(v_0)$ Re-writing the above equation, we get: $...
2
votes
2answers
48 views

Are mini batches sampled randomly in Keras' Sequential.fit method()

When you .fit a Keras Sequential() model, you can specify a batch_size parameter. I have ...
1
vote
3answers
52 views

How do GD, Batch GD, SGD, and Mini-Batch SGD differ?

How do these four types of gradient descent functions differ from each other? GD Batch GD SGD Mini-Batch SGD
2
votes
1answer
27 views

Intractability in Variational Autoencoders

I'm having difficulty understanding when integrals are intractable in variational inference problems. In a variational autoencoder with observation $x$ and latent variable $z$ we want to maximize ...
0
votes
0answers
14 views

Do we include the L2 regularization loss gradients when visualizing the norm of the gradients?

During training I need to plot the gradient norms at each layer to monitor the progress. When the loss function is made up of the main loss term plus the L2 regularization term, should we only plot ...
0
votes
0answers
18 views

Can one set manual adaptive learning in SGDRegressor()?

I wanted to update learning rate $r = r/2$ in each iteration of SGDRegressor(). I cannot find any way so far to update the learning rate manually. There is a choice called adaptive but it doesn't look ...
0
votes
1answer
29 views

Compute gradients in parallel

Here is part of my code: ...
0
votes
0answers
7 views

Calculate gradients in parallel

Here is part of my code: ...
0
votes
0answers
18 views

Can optimal intercept be gotten from optimal hyperplane in gradient descent?

I have read if $w_1^T$ is a plane and $b$ be intercept on $y$ axis. Then, $w_1^T.x_1+b=0$ if $w_1,x_1 \in R^{d}$ in $d$ dimensional hyperplane. Alternatively, $w^Tx =0$ if $w=<b, w_1>$ & $x=&...
2
votes
0answers
19 views

Get derivatives from your NN

How can I get the gradient of a node in the NN with respect to another one? I need to train a NN, which for the sake of simplicity has 2 neurons as input (x, y), a neuron as a bottleneck (z), and 2 ...
1
vote
1answer
16 views

How Stochastic Gradient Descent used like Mini Batch gradient descent?

As I know Gradient Descent has three variants which are: 1- Batch Gradient Descent: processes all the training examples for each iteration of gradient descent. 2- Stochastic Gradient Descent: ...
0
votes
0answers
17 views

How to create an autograd library from scratch like pytorch?

I am trying to implement a deep learning library from scratch. Most common DL framework uses autograd. Unfortunately, I haven't seen a lot of resources on how to create one autograd library. Is there ...
0
votes
0answers
25 views

Calculate Gradient in Machine Learning Classification Problem

I going through the optimization techniques in DL. I went through a code snippet of calculating the gradient. ...
1
vote
0answers
35 views

CartPole v1 - Simple backprop with 1 hidden layer

I'm trying to solve the CartPole-v1 problem from OpenAI by using backprop on a one-layer neural network - while updating the model at every time step using State action values (Q(s,a)). I'm unable to ...
1
vote
1answer
22 views

Why does my linear regression model converge to a non-zero gradient value?

I have a basic 2D Linear Regression model coded out (using gradient descent), yet it doesn't seem to work as well as it should. What I expect is that m and ...
1
vote
1answer
62 views

Am I using optim.SGD incorrectly in pytorch?

I am doing reinforcement learning in checkers. After each game the network beats itself, I calculate the loss of every individual position in the game, call backward(), and step(). I am beginning to ...
0
votes
0answers
13 views

What's the correct way of implementation of cost function and gradient function in logistical regression after regularisation?

This is the cost function of logistic regression: which i could implement correctly, with the code : ...
0
votes
0answers
16 views

Gradient of batchnorm layer

In recent paper about How Does Batch Normalization Help Optimization? by Satunkar et.al. The paper mentions facts about the derivative of a loss function through a batchnorm layer. The paper state ...
0
votes
0answers
20 views

my gradient decent isn't working right

i'm very new to machine learning, i was doing an exercise about classification using the sigmoid as hypothesis, i don't know what's wrong but my cost function keep increasing and the slope of the cost ...
4
votes
0answers
51 views

Gradient Descent in ReLU Neural Network

I’m new to machine learning and recently facing a problem on back propagation of training a neural network using ReLU activation function shown in the figure. My problem is to update the weights ...
1
vote
2answers
141 views

which metric is better for boosting methods

I work on a dataset of 300 000 samples and I try to make a comparison between logistic regression (with gradients descent) and a LightBoost for binary classification in order to choose the better one. ...
0
votes
0answers
17 views

How to retain dependency between variables in PyTorch?

I am modeling k-dimensional positions over time t = 0...T using a set of initial positions Z0 with requires_grad=True and storing the results in Z with requires_grad=False for the remaining T-1 time ...
1
vote
0answers
73 views
1
vote
1answer
39 views

Neural network back propagation gradient descent calculus

So I've drawn a neural network diagram below: where $x_1, x_2,\ldots,x_m$ are the input layer, $h_1, h_2$ are the hidden layer and $\hat y_1, \hat y_2,\ldots \hat y_k$ are the output layer. In the $W^...
1
vote
1answer
43 views

Gradient descent with infinite gradient value

Given a function $f(x)$ and $\frac{\partial f(x)}{\partial x_i}=\frac{f^2(x1,...,x_i+\pi/2,...,x_n)-f^2(x1,...,x_i-\pi/2,...,x_n)}{f(x)}$. When $f(x)\to0$, $\frac{\partial f(x)}{\partial x_i}$ could ...
1
vote
2answers
44 views

Gradient Descent Convergence

I'm a double major in Math and CS interested in Machine Learning. I'm currently taking the popular Coursera course by Prof. Andrew. He's talking and explaining Gradient Descent but I can't avoid ...
2
votes
2answers
47 views

Calculating derivative of error at point x with respects to weight w_j

I don't know how the equation below goes from line 2 to 3 after the derivative term is moved inside the brackets. Specifically, how is it calculating the derivative of log(y_hat)? Also, if anyone ...
1
vote
1answer
15 views

Partial derviative of prediction (sigmoid applied) with respect to weight

I am very confused as to where a seemingly "extra" term is included in the above mentioned calculation in my Udacity course. The above is taking the derivative of a sigmoid so why isn't it just $$...
1
vote
1answer
60 views

Understanding general approach to updating optimization function parameters

This question not related to a specific method or technique, rather there is a broader concept that I'm struggling to see clearly. Introduction In machine learning, we have loss functions that we're ...
13
votes
4answers
3k views

Is Gradient Descent central to every optimizer?

I want to know whether Gradient descent is the main algorithm used in optimizers like Adam, Adagrad, RMSProp and several other optimizers.
1
vote
1answer
72 views

Linear classifier and gradient descent

I understood that gradient descent is needed to find the local extremum of any function, but how is it applied to the linear classifier (single matrix, two classes case, for example)? How does it step ...
0
votes
0answers
11 views

Nesterov Momentum update equation

I am currently taking CS231n. In the lecture, training neural networks part 2, Nesterov momentum is introduced where I understood theoretically that the gradient is calculated at a later point where ...
0
votes
1answer
18 views

Can we use decreasing step size to replace mini-batch in SGD?

As far as I know, mini-batch can be used to reduce the variance of the gradient, but I am also considering if we can achieve the same result if we use the decreasing step size and only single sample ...
0
votes
0answers
21 views

Do 3D-CNN suffer from exploding and vanishing gradient?

I am building a video classification network, I see 2 options for the same CNN+LSTM and 3D-CNNs, Do 3D-CNNs suffers from exploding and vanishing gradients like LSTMs or are 3d-CNN better at handling ...
1
vote
0answers
28 views

Modelling a startup's funding journey with Brownian Motion

I am trying to implement a "light" version of a paper (Hunter, Saini & Saman 2017), in which the authors build a model capable of predicting the probability that a startup will exit (either by ...
0
votes
2answers
30 views

Weight training breakdown in machine learning

I'm not sure if this exists. Is there such a situation where weights in gradient descent fail to work or break up? If so, how and when?
1
vote
1answer
26 views

Where is my error in understanding gradient descent calculated two different ways?

The gradient descent algorithm is, most simply, w'(i) = w(i)-r*dC/dw(i) where w(i) are the old weights, w'(i) are the new weights, C is the cost, r is the learning rate. I'm aware of the graphical ...
1
vote
0answers
49 views

How to calculate the gradient for nce_loss in tensorflow

I need to calculate the gradient of a tensorflow that is stored. I can restore the graph and weights using: ...
2
votes
2answers
72 views

Constant Learning Rate for Gradient Decent

Given, we have a learning rate, $\alpha_n$ for the $n^{th}$ step of the gradient descent process. What would be the impact of using a constant value for $\alpha_n$ in gradient descent?