Questions tagged [gradient-descent]

Gradient Descent is an algorithm for finding the minimum of a function. It iteratively calculates partial derivatives (gradients) of the function and descends in steps proportional to those partial derivatives. One major application of Gradient Descent is fitting a parameterized model to a set of data: the function to be minimized is an error function for the model.

Filter by
Sorted by
Tagged with
-3 votes
0 answers
42 views
+50

Verifying my understanding of MLE & Gradient Descent in Logistic Regression

Here is my understanding of the relation between MLE & Gradient Descent in Logistic Regression. Please correct me if I'm wrong: 1) MLE estimates optimal parameters by taking the partial derivative ...
user avatar
  • 315
0 votes
0 answers
11 views

How do you find the eigenvalues of the matrix for the following momentum gradient descent?

The following question is based purely on the material available on MIT's open courseware youtube channel. (https://www.youtube.com/watch?v=wrEcHhoJxjM). In it, Professor Gilbert Strang explains the ...
user avatar
0 votes
0 answers
20 views

ResNet: Derive the gradient matrices w.r.t. W1 and W2 and backprop equation in a Residual Network

How would I go about step by step deriving stochastic gradient matrices w.r.t. W1 and W2 and backpropagation equation in a residual block that is a part of a larger ResNet network with forward ...
user avatar
  • 1
0 votes
0 answers
13 views

From what function do come the gradients that I use to adjust weights?

I have a question about the loss function and the gradient. So I'm following the fastai (https://github.com/fastai/fastbook) course and at the end of 4th chapter, I got myself wondering. From what ...
user avatar
2 votes
2 answers
28 views

Difference between OLS and Gradient Descent in Linear Regression

I understand what Ordinary Least Squares and Gradient Descent do but I am just confused about the difference between them. The only difference I can think of are- Gradient Descent is iterative while ...
user avatar
  • 121
1 vote
0 answers
22 views

Need help to understand the formula of gradient descent with multiple features

I am trying to implement gradient descent with multiple features after listening to Andrew Ng's Coursera lecture. gradient descent for multiple features So for example when calculating for theta 1, ...
user avatar
  • 11
0 votes
0 answers
5 views

What is "Gradient × Hidden States" explainability method? Is there any documentation about it?

I am doing a literature review on post-hoc explainability methods based on gradient. I stumbled upon one I didn't heard of to extract highlights from a trained model in this post-hoc fashion: We ...
user avatar
1 vote
0 answers
7 views

How to interpret integrated gradients in an NLP toxic text classification use-case?

I am trying to understand how integrated gradients work in the NLP case. Let $F: \mathbb{R}^{n} \rightarrow[0,1]$ a function representing a neural network, $x \in \mathbb{R}^{n}$ an input and $x' \in ...
user avatar
1 vote
0 answers
14 views

Understanding Learning Rate in depth

I am trying to understand why the learning rate does not work universally. I have two different data sets and have tested out three learning rates 0.001 ,0.01 and 0.1 . For the first data set, I was ...
user avatar
  • 11
0 votes
0 answers
11 views

Is there a difference between AutoGrad and explicit derivatives (gradient)?

Will there be some differences between applying AutoGrad on the loss function (using a python library) and applying explicit gradient (the gradient from the paper or the update rule)? For example: ...
user avatar
  • 1
0 votes
1 answer
29 views

Why does neural network need loss as scalar?

I have a loss function that's a weighted cross entropy loss for binary classification ...
user avatar
0 votes
0 answers
9 views

Is every timestep when training an LSTM part of the error, or just the end of the sequence?

I'm attempting to write my own LSTM network in C++ for fun. I've already got a regressive and classification network working with regular perceptrons and it works well. What I do currently is divide ...
user avatar
1 vote
0 answers
17 views

Why it's better to use double floating point precision instead of single point precision for gradient check

I came across the following," A common pitfall is using single precision floating point to compute gradient check. It is often that case that you might get high relative errors (as high as 1e-2) ...
user avatar
  • 179
0 votes
0 answers
56 views

Training error vs testing error for batch gradient descent on Pyrhon, can't understand what I'm supposed to do

I have this dataset containing training data and testing data , and I have to plot training and testing errors for the gradient descent algorithm with square and logistic losses. I'm a beginner, I'm a ...
user avatar
0 votes
0 answers
11 views

Gradient descent to minimize the sum of arc length distances on a unit sphere

I have to write a batch gradient descent algorithm to find the point on the unit sphere (in any number of dimensions) which minimizes the sum of arc length distances to each of a set of given points ...
user avatar
0 votes
0 answers
25 views

How does MultiVariate Linear Regression actually calculate each coefficient?

W​hile calculating the coefficients in gradient descent for multivariate linear regression, the loss accounted in all the coefficients(parameters) updating-equations is the same, loss=predicted-y x_1=...
user avatar
0 votes
1 answer
19 views

Loss-value of normal equation vs gradient descent

My question is if gradient descent can give a better aproximation than normal equation in Python? for the Loss function, I wrote ...
user avatar
0 votes
1 answer
19 views

Vanishing gradients: examine output gradients

For a feedforward network or RNN, in theory we should examine the output gradients with respect to the weights over time to check whether it vanishes to zero. In my code below I am not sure whether it ...
user avatar
  • 113
0 votes
0 answers
33 views

Adding a group specific penalty to binary cross-entropy

I want to implement a custom Keras loss function that consists of plain binary cross-entropy plus a penalty that increases the loss for false negatives from one class (each observation can belong to ...
user avatar
  • 1
1 vote
2 answers
113 views

MLE & Gradient Descent in Logistic Regression

In Logistic Regression, MLE is used to develop a mathematical function to estimate the model parameters, optimization techniques like Gradient Descent are used to solve this function. Can somebody ...
user avatar
  • 315
0 votes
0 answers
21 views

Question from a paper: I do not understand why it is stated that SGD employs the bootstrapping to calculate gradient?

In this paper, they state that: As SGD employs the bootstrapping (i.e., random sampling with replacement) [67] for gradient calculation, we can obtain the unbiased estimation of standard gradients ...
user avatar
  • 121
1 vote
2 answers
232 views

Relation between MLE (Maximum Likelihood Estimation) & Gradient Descent

What are the similarities & dissimilarities between MLE (used to find the best parameters in logistic regression) & Gradient Descent?
user avatar
  • 315
0 votes
0 answers
7 views

How to overcome extremely variable model performance?

I am training an LSTM and the model performance seems to range from near perfect to dreadful by visual inspection of predicted values - it seems by random initialisation simply retraining the model ...
user avatar
1 vote
1 answer
105 views

Gradient descent implementation of logistic regression

Objective Seeking for help, advise why the gradient descent implementation does not work below. Background Working on the task below to implement the logistic regression. Gradient descent Derived the ...
user avatar
  • 463
1 vote
0 answers
11 views

How do I find the gradients of an X -> RELU -> RELU -> Softmax network?

I am trying to use the MNIST digits dataset with logistic regression. I am only using numpy as I would like to implement a simple project myself before using something like pytorch. My network ...
user avatar
  • 111
1 vote
0 answers
39 views

TypeError: only size-1 arrays can be converted to Python scalars

I am doing multiple linear regression using gradient descent and am getting this type-error ...
user avatar
0 votes
0 answers
30 views

What stochastic gradient descent is and isn't

I've read what stochastic gradient descent is in multiple different places but everyone describes it in a slightly different way which makes me questions what is really is. Is my definition below ...
user avatar
  • 101
1 vote
1 answer
52 views

Why does stochastic gradient descent lead us to a minimum at all?

Why do we think that stochastic gradient descent is going to find a minimum at all? I mean on each iteration SGD moves in the direction that reduces only current batch's error (SGD doesn't care about ...
user avatar
  • 121
2 votes
1 answer
163 views

Does gradient descent always find global minimum for specific regression type?

From my understanding, linear regression is used for predicting an output based on an input using a linear equation that is optimally fitted to some input data. We choose the best fitted linear ...
user avatar
  • 23
0 votes
0 answers
12 views

Formula of momentum gradient descent optimizer

Learning about the optimizers recently, I was confused about the formula for momentum. I mean, I understood the concept but I came across the following 2 formulas while learning. I see that the left ...
user avatar
  • 21
0 votes
2 answers
100 views

How does Gradient Descent work? [duplicate]

I know the calculus and the famous hill and valley analogy (so to say) of gradient descent. However, I find the update rule of the weights and biases quite terrible. Let's say we have a couple of ...
user avatar
0 votes
1 answer
25 views

Understanding intution behind sigmoid curve in the context of back propagation

I was trying to understand significance of S-shape of sigmoid / logistic function. The slope/derivative of sigmoid approaches zero for very large and very small input values. That is $σ'(z) ≈ 0$ for $...
user avatar
  • 165
0 votes
0 answers
16 views

RMSprop in weight update - what if vertical slopes small and horizontal slopes large?

I have a question regarding the intuition behind RMSprop, As shown in the lecture video of Deep Learning Specialization by Andrew Ng, RMSprop helps to reduce the oscillation (the values of the ...
user avatar
1 vote
0 answers
33 views

Is saddle point a cause for the vanishing gradient problem

I am a beginner to neural networks and I am writing a report summarising on the causes and solutions to the vanishing gradient problem. From what I have read, the 2 main causes are the repeated ...
user avatar
  • 113
0 votes
0 answers
11 views

cost function diverging in batch gradient descent

I am trying to implement the gradient descent method in python. I would like the calculation to stop when abs(J-J_new) reaches a certain tolerance level (i.e. it ...
user avatar
0 votes
0 answers
32 views

Gradient descent in linear regression converges but the trend line is incorrect

For the dataset https://physics.info/linear-regression/dash-world.txt, I have been trying to implement linear regression for predicting the men record times as a function of year. I have used gradient ...
user avatar
0 votes
0 answers
19 views

Matrix multiplication

I have downstream gradient for every $sample$ (each row for every $x_i$) $$ \begin{bmatrix} 0.0062123 & -0.00360166 & -0.00479891 \\ -0.01928449 & 0.01240768 & 0.01493274 \\ ...
user avatar
0 votes
0 answers
32 views

How to check for vanishing gradient in CNN?

Is there is a way to check for vanishing gradient for CNNs using Keras? Like, for example, drawing the weight distributions for each layer and seeing if they are vanishing to zero?
user avatar
  • 7
0 votes
0 answers
20 views

Andrew Ng Deep Learning Gradient Descent of Softmax is just y_hat - y?

At about 8:30 in the video here: https://www.youtube.com/watch?v=ueO_Ph0Pyqk so for the given example with 4 classes and first ground truth y being [0,1,0,0] and y_hat being [0.3,0.2,0.1,0.4] for ...
user avatar
0 votes
1 answer
24 views

Linear regression and gradient descend equations

I'm pretty new to ML and was starting out with linear regression combined with gradient descend. This is the equation I was trying to achieve using javascript- And this is what I came up with in js- <...
user avatar
1 vote
2 answers
61 views

How can the ReLU function lead to convergence?

The gradient descent algorithm is based on the fact that the gradient decreases as we move towards the optimum point. However, in the activations by the ReLU ...
user avatar
1 vote
2 answers
140 views

Vanishing gradient problem even after existence of ReLu function?

Let's say I have a deep neural network with 50 hidden layers and at each neuron of hidden layer the ReLu activation function is used. My question is Is it possible for vanishing gradient problem to ...
user avatar
  • 21
0 votes
0 answers
24 views

Neural Net gradient descend

I was planning on making my own neural network library in C++ and was going through other's code to make sure I am on right track. Below is a sample code that I am trying to learn from. Everything in ...
user avatar
0 votes
0 answers
10 views

Compare rate of change for multiple object/weights

For a Neural Network, the weight update equation is: However, there are millions of such weights W_i. If I am interested in capturing how much each weight/connection W_i is changing as compared to ...
user avatar
  • 121
0 votes
0 answers
39 views

Stochastic Gradient Region of Confusion

I have come across the following diagram which explains the behavior of SGD graphically. Based on this graphical representation, the gradient of the individual data tend to fluctuate more when it ...
user avatar
0 votes
1 answer
12 views

Gradient descent different implementation cause error

We know that we can get closer to the local minimum of the function by descending our argument according to that rule $$w1 = w0 − γ∇f$$ For example I have a linear regression model that depends on $b,...
user avatar
1 vote
1 answer
35 views

Gradient descent method

If we suppose that this is formula for gradient descent method: $$x_{n+1}=x_n-\lambda\cdot{{df(x)}\over{dx}},\ n=0,1,2,3,...$$ Since there is no exact value that we subtract instead of derivative, ...
user avatar
0 votes
1 answer
116 views

Why the gradient of a ReLU for X>0 is 1?

Gradient is derivative of several variables. I can't understand why is the gradient of a ReLU for X>0 is 1 ? and 0 for x < 0 ? I tried to search for proof and examples but didn't found any good ...
user avatar
  • 267
0 votes
0 answers
16 views

Step size finds by quadratic fitting in steepest descent

I have a function $f=(1-x_1)^2 + (x_2-(x_1^2))^2$ and initial point $[0,5]$. I wonder how I will find step size by quadratic fitting using the (e.g. $0.01$) value in Steepest Descent with Matlab. To ...
user avatar
  • 1
0 votes
0 answers
17 views

Bug in single layer Adaline Neuron implementation

I am trying to implement a single layer Adaline neuron, with the following mathematical foundation: The cost function is defined as: The weight update is defined as: inserting the partial derivative ...
user avatar
  • 101

1
2 3 4 5
9