Podcast #128: We chat with Kent C Dodds about why he loves React and discuss what life was like in the dark days before Git. Listen now.

Questions tagged [gradient-descent]

Gradient Descent is an algorithm for finding the minimum of a function. It iteratively calculates partial derivatives (gradients) of the function and descends in steps proportional to those partial derivatives. One major application of Gradient Descent is fitting a parameterized model to a set of data: the function to be minimized is an error function for the model.

Filter by
Sorted by
Tagged with
1
vote
0answers
18 views

How can I regularize the output of a layer from scratch (without using Keras)?

I am trying to build a Convolutional Neural Network after reading notes from Stanford's cs231n course. I use ELU activation as activation function, and SoftMax as my classifier. Architecture is simple:...
2
votes
1answer
27 views

Why in this case are gradient steps not perpendicular to contour lines?

There is a theorem that gradient at point is perpendicular to tangent line to contour line at given point. Why in this picture it seems that this rule is not respected? source: http://www....
0
votes
0answers
6 views

SGDClassifier partial_fit() for online learning - is one step of gradient descent enough?

I'm interested in incremental (online) learning for my logistic regression model trained with SGDClassifier. Basically updating the model as more labeled data comes ...
2
votes
1answer
44 views

Calculating the average of gradient decent

I am currently studying the backpropagation process and gradient decent algorithm form the book Neural Networks and Deep Learning written by Michael Nielsen and 3Blue1Brown channel in YouTube. My ...
0
votes
1answer
18 views

Cannot fig out error in my gradient function implementation in python

Im trying to implement following gradient descent function in Python for logistic regression: $∇θ(−logL)=−X^T 􏰀(y−e^{Xθ}􏰁)$ This is my python implementation: ...
0
votes
0answers
19 views

Gradient equations of gaussian kernel discriminant trained with gradiant descent

I am having a hard time trying to find the gradient equations for the weight $\alpha^t$ and $w_0$ for a gaussian kernel discriminant trained with gradient descent with the following error function $$E(...
0
votes
0answers
12 views

How is this function (for updating a Stochastic Gradient Descent model) called without a parameter?

I'm in the middle of a Deep Learning Course offered by DataCamp and the example below was given for optimizing a SGD model: As you can see, the function "get_new_model" requires one parameter: "...
0
votes
1answer
23 views

Back-propagation and stochastic gradient descent

Is backpropagation a learning method or an optimisation method? How are backpropagation and stochastic gradient descent related to each other?
3
votes
1answer
69 views

Grid search or gradient descent?

Assume we have a neural network and one if its activation functions is a function of parameter a. We want to find the weights and parameter a that leads to the minimum loss on the validation set which ...
0
votes
1answer
27 views

gradient descent for non convex function like $-x^2$

I know how to calculate gradient descent for a convex function where there is only one global minima. Also, I know methods to handle cases where the function is a non-convex function. What is really ...
1
vote
0answers
14 views

What is the difference between these 2 training scenarios?

I have a very large dataset and due to computational constraints, I have to divide the data into 20 parts (each part is around 1.5GB). I constructed a deep CNN model using Keras for this dataset. The ...
0
votes
2answers
24 views

How to form and minimise custom features for classification in supervised learning

I am having an issue in understanding how to form the features based on particular math formula, and how to adjust the weights with. The aim is to draw ellipses for each unique category of points. ...
0
votes
2answers
89 views

What is the difference between gradient descent and gradient boosting? Are they interdependent on each other by any way?

What is the difference between gradient descent and gradient boosting? Are they interdependent on each other in any way ?
0
votes
1answer
25 views

Similarity of XGBoost models?

Is xgboost with n_estimators = 100 and learning_rate = 0.1, same as xgboost with n_estimators = 50 and learning_rate = 0.2 ?
0
votes
0answers
18 views

Proof that gradient descent takes exponential time for escaping saddle points

https://arxiv.org/pdf/1705.10412.pdf I was going through this paper, and understood the crux of it. But in appendix, the complete proof of it is given which was a bit tough for me mathematically. So ...
0
votes
0answers
8 views

Purpose of gamma multiplier in gradient boosting

looking through the mathematics of gradient boosting on the relevant wikipedia page, intuitively what is the purpose of the multiplier $\gamma_i$? This term does not appear in the following ...
4
votes
1answer
33 views

How are weight updates handled in Batch Gradient Descent vs SGD?

My current understanding is that in SGD, after each data sample, the loss is used to update each weight. Ex: With 1000 samples and a network with 10 weights, there will be 10,000 individual weight ...
0
votes
0answers
12 views

Theano gradient descent and cost function issue

Sorry but can anyone point out whats wrong with this code? ...
2
votes
0answers
30 views

Linear Regression in Python using gradient descent

I am trying to implement a simple multivariate linear regression model without using any inbuilt machine libraries. So far, I have been able to get a root mean squared error for training about $2.93$ ...
0
votes
2answers
67 views

Gradient Descent or Normal Equation?

Hi guys I am really struggling with this question. I need to pick the correct choice: Suppose you have a dataset with m = 50 examples and n = 15 features for each example. You want to use ...
0
votes
2answers
74 views

What's the proper way to do back propagation in Deep Fully Connected Neural Network for binary classification

I tried to implement a Deep fully connected neural network for binary classification using python and numpy and used Gradient Descent as optimization algorithm. ...
1
vote
3answers
64 views

Gradient descent in a noisy environment

How to know the right direction in a noisy environment? In the typical example of neural network learning, we can see several local minima. The gradient descent is choosing one local minimum and ...
0
votes
0answers
22 views

problem with vanishing/exploding gradient problems solution

I have few doubts around vanishing/exploding gradients. The problem with vanishing gradient is, When the weights are randomly initialized in a deep network, During back propagation initial layers ...
2
votes
1answer
33 views

SGD, calculating it by hand

While I find a lot of material of SGD (Stochastic gradient descent), I am struggling to find one concrete example with numbers e.g. calculating it by hand for let's say, one iteration would help me a ...
2
votes
1answer
111 views

Gradient descent formula

I came across an interesting book about neural network basics, and the formula for gradient descent from one of the first chapters says: Gradient descent: For each layer update the weights ...
0
votes
0answers
7 views

Backtracking Line search for Multiclass classification gradient descent

For my case i am dealing with multiclass problem and there are total 28 direction component for each class and there are total 5 classes, for given equation above, f(w+nd) and f(w) gives scaler values ...
3
votes
1answer
91 views

Fast Python implementation of the gradient descent

I'm looking for fast Python implémentations of gradient descent optimization algorithm. I have a convex problem , with no constraint, so for now I'm using the BFGS algorithm implemented in scikit-...
1
vote
0answers
129 views

SKLearn Boston dataset gradient descent not working

I am trying to compare some simple methods for linear regression as an exercise. I have already used LinearRegression from the SKLearn library in python as well as the formula of linear regression. ...
2
votes
1answer
29 views

Quadratic approximation of L1 regularized cost function

I'm reading the Deep Learning book of Goodfellow, but I fail to see why minimization of (7.22) gives (7.23). I tried to compute the gradient w.r.t. the $w_{i}$ and set this to zero, but it doesn't ...
4
votes
2answers
43 views

Question of using gradient descent instead of calculus. I checked previous questions there are still points to clarify

First of all I checked http://stats.stackexchange.com/questions/23128/solving-for-regression-parameters-in-closed-form-vs-gradient-descent, http://stackoverflow.com/questions/26804656/why-do-we-use-...
6
votes
2answers
122 views

Why is taking the gradient of the average error in SGD not correct, but rather the average of the gradients of single errors?

I am a little confused about taking averages in cost functions and SGD. So far I always thought in SGD you would compute the average error for a batch and then backpropagate it. But then I was told in ...
0
votes
0answers
25 views

Does a max-pooling layer in a ConvNet contribute to the “vanishing gradient” problem?

I would answer no, but am not sure if I'm missing something and hope you can help me out: The derivative of a max-pooling layer in a ConvNet is one w.r.t. the maximum value and zero for all others. A ...
1
vote
0answers
39 views

Gradient boosting, where did the constant go?

In the very early papers on gradient boosting, the ensemble would include a constant and a sum of base learners i.e. $F(X) = a_0 + \sum\limits_{i} a_i f_i(X)$ The constant is fitted first (i.e. if ...
3
votes
2answers
98 views

Why does feature scaling improve the convergence speed for gradient descent?

From this article, it says: We can speed up gradient descent by scaling. This is because θ will descend quickly on small ranges and slowly on large ranges, and so will oscillate inefficiently down ...
0
votes
1answer
29 views

Derivative of activation function used in gradient descent algorithms

Why is it necessary to calculate the derivative of activation functions while updating model( regression or NN) parameters? Why is the constant gradient of linear functions considered as a ...
0
votes
1answer
25 views

Confused with the derivation of the gradient descent update rule

I have been going over some theory for gradient descent. The source I am looking at said that the change in cost can be described by the following equation: $$∆C=∇C∙∆w$$ where $∇C$ is the gradient ...
1
vote
1answer
20 views

Pytorch - Gradient distribution between functions

https://colab.research.google.com/github/pytorch/tutorials/blob/gh-pages/_downloads/neural_networks_tutorial.ipynb Hi I am trying to understand the NN with pytorch. I have doubts in gradient ...
1
vote
0answers
12 views

Why does NAG cause unstable validation loss?

I'm building a neural network for a classification problem. When playing around with some hyperparameters, I was surprised to see that using Nesterov's Accelerated Gradient instead of vanilla SGD ...
0
votes
1answer
18 views

What does this expression from gradient descent mean?

I am looking over some neural network theory and came across this equation, coupled with this description (gradient descent ball-valley analogy): ''let's think about what happens when we move the ...
1
vote
3answers
62 views

Different learning rates for each dimension

I have been thinking about why normalization and scaling are done for each feature in the basic context of gradient descent. One thing that got me wondering is that we use a pre-defined set of ...
0
votes
1answer
36 views

Does it make sense to train an Autoencoder for Dimensionality Reduction using Mini-Batch Gradient Descent?

I want to reduce the dimensionality of a dataset using a stacked Autoencoder. The size of the dataset and the computing power at my disposal make it very difficult to train the Network using simple, ...
1
vote
0answers
99 views

Updating Weight Using Updates on Related Data

Suppose $$ x=Ay $$ The $x$ is $M\times 1$, $y$ is $N \times 1$ and $A$ is $M\times N$ We have the data $x$ and would like to know what $y$ is. However, the matrix $A$ is too large for pseudo-...
0
votes
0answers
39 views

In an RNN, if the gradients don't vanish for long/distant terms, won't the derivative of the error be either divergent to infinity or oscillatory?

P.S. Crosss posted here- https://stats.stackexchange.com/questions/413843/in-an-rnn-if-the-gradients-dont-vanish-for-long-distant-terms-wont-the-deriv, as I've got no answer, I'm asking here: In my ...
0
votes
0answers
36 views

How to get the weights of a linear model by solving normal equation?

In chapter 6.1 of the book Deep Learning, the author tries to learn the XOR function by using a linear model (on page 168). Linear Model: $f(\mathbf{x};\mathbf{w},b)=\mathbf{x}^T\mathbf{w}+b$ MSE ...
2
votes
1answer
81 views

Dueling Network gradient with respect to Advantage stream

Looking at Dueling DQN: $Q = V + A - mean(A)$ For simplicity, let's assume we are working with 4 neurons. Recall that Value stream only has 1 neuron $(v_0)$ Re-writing the above equation, we get: $...
2
votes
2answers
233 views

Are mini batches sampled randomly in Keras' Sequential.fit method()

When you .fit a Keras Sequential() model, you can specify a batch_size parameter. I have ...
1
vote
3answers
446 views

How do GD, Batch GD, SGD, and Mini-Batch SGD differ?

How do these four types of gradient descent functions differ from each other? GD Batch GD SGD Mini-Batch SGD
2
votes
1answer
57 views

Intractability in Variational Autoencoders

I'm having difficulty understanding when integrals are intractable in variational inference problems. In a variational autoencoder with observation $x$ and latent variable $z$ we want to maximize ...
0
votes
0answers
27 views

Do we include the L2 regularization loss gradients when visualizing the norm of the gradients?

During training I need to plot the gradient norms at each layer to monitor the progress. When the loss function is made up of the main loss term plus the L2 regularization term, should we only plot ...
0
votes
0answers
30 views

Can one set manual adaptive learning in SGDRegressor()?

I wanted to update learning rate $r = r/2$ in each iteration of SGDRegressor(). I cannot find any way so far to update the learning rate manually. There is a choice called adaptive but it doesn't look ...