Questions tagged [gradient-descent]

Gradient Descent is an algorithm for finding the minimum of a function. It iteratively calculates partial derivatives (gradients) of the function and descends in steps proportional to those partial derivatives. One major application of Gradient Descent is fitting a parameterized model to a set of data: the function to be minimized is an error function for the model.

Filter by
Sorted by
Tagged with
1
vote
1answer
51 views

How to find slope of curve at certain points

how to find slope at certain points circled in blue in below curve ? Are these below 2 approaches valid ? though they give different results . How to automatically find the points where the slope ...
0
votes
0answers
24 views

Multivariable linear gradient descent resulting in inf

I am trying to implement a multivariable gradient descent algorithm, it seems to start working fine, and works on smaller datasets, but applying it to larger datasets the variables overflow and cause ...
1
vote
1answer
42 views

Why do we only care about convex functions when doing Gradient Descent/SGD?

I mean I know why we specifically care about convex functions: it's because their local minimum are also global, and so you just have to "follow a path which goes down" to find the minima of ...
0
votes
1answer
68 views

gradient descent for non convex function like $-x^2$

I know how to calculate gradient descent for a convex function where there is only one global minima. Also, I know methods to handle cases where the function is a non-convex function. What is really ...
2
votes
1answer
110 views

Linear Regression in Python using gradient descent

I am trying to implement a simple multivariate linear regression model without using any inbuilt machine libraries. So far, I have been able to get a root mean squared error for training about $2.93$ ...
-1
votes
3answers
38 views

Gradient descent does not converge in some runs and converges in other runs in the following simple Keras network

When training a simple Keras NN (1 input, 1 level with 1 unit for a regression task) during some runs I get big constant loss that does not change in 80 batches. During other runs it decreases. What ...
3
votes
2answers
91 views

Getting NN weights for every batch / epoch from Keras model

I am trying to get weights for every batch / epoch from Keras model after it is trained. To do so I use callback to make model save weights during training. Yet after model is trained it looks like I ...
0
votes
1answer
23 views

Why use gradient descent on Deep Nets / RNNs when cost function is not convex?

Why do we use gradient descent on very non-convex loss functions such as in Deep nets / RNNs rather than a heuristic search (genetic algorithms, simulated annealing, etc)?
1
vote
1answer
138 views

What is the significance of underflow during parameter update using stochastic gradient descent?

Background I am using scikit-learn's MLPRegressor to learn a model with the following arguments: ...
1
vote
0answers
29 views

SVM with gradient descent

The constrained optimization problem in SVM is given by min 1/2 ||w||^2 s.t y(i)(w^T x(i) + b >= 1 for all i Now converting this to an unconstrained optimization problem gives the lagriangian L as ...
1
vote
1answer
38 views

Why standard distribution for ML [closed]

Data normalization: It ensures that each input (each pixel value, in this case) comes from a standard distribution. This standardization makes our model train and reach a minimum error, faster! my ...
3
votes
2answers
140 views

How does the descending gradient know what weights to adjust?

I was reading about descending gradient. How does the descending gradient know what weights to adjust? Does it adjust to all network weights at the same time? Does each weight have an associated error?...
2
votes
3answers
34 views

Neural Network Loss Function - Mean Square Error: questions about what 'n' signifies

I'm very new to neural networks and have recently learnt about the loss functions used with neural networks. This question is in regards to the mean square error metric, defined as (from the textbook ...
0
votes
1answer
44 views

Understanding the concept vanishing gradient and exploding gradient problem in terms of training data

I'm trying to figure out the essence of the concepts "vanishing gradient and exploding gradient problem" in terms of real-world input-output training examples instead of in terms of the properties of ...
1
vote
1answer
13 views

Maximum Likelihood with Gradient Descent or Coordinate Descent blows up

Context The maximum likelihood estimators for a Normal distribution with unknown mean and unknown variance are $$ \widehat{\mu} = \frac{1}{n}\sum_{i=1}^n x_i \qquad \text{and} \qquad \widehat{\sigma}^...
0
votes
1answer
27 views

Neural Network not learning when more than 1 training data is given

I am very new to neural networks and data science in general and wanted to try getting my hand in making a simple neural network in python. I tried to make a neural network from scratch hoping to ...
2
votes
1answer
388 views

Intractability in Variational Autoencoders

I'm having difficulty understanding when integrals are intractable in variational inference problems. In a variational autoencoder with observation $x$ and latent variable $z$ we want to maximize data ...
1
vote
1answer
57 views

Why does Siamese neural networks use tied weights and how do they work?

Reading this paper on one-shot learning "Siamese Neural Networks for One-shot Image Recognition" I was introduced to the idea of Siamese Neural Networks. What I did not fully grasp was what they ...
2
votes
2answers
86 views

How does a zero centered activation functions like tanh helps in gradient decent?

I know that, if X are all positive, or negative then the sign of the downstream gradient will be same as that of the upstream gradient, but what I don't understand is how the zero centered activation ...
3
votes
0answers
70 views

Why the sigmoid activation function results in sub-optimal gradient descent?

I need some help understanding the second shortcoming of the sigmoid activation function as described in this video from Stanford. She says that because the output of sigmoid is always positive, that ...
-1
votes
0answers
15 views

How do I fit a custom curve to a mathematical model using TensorFlow/Keras?

I'm trying to fit a simple S.I.R mathematical model to the U.K. COVID dataset using TensorFlow/Keras. I've spent days looking to no avail. I have the data from a csv file and stored in a tx1 NumPy ...
4
votes
2answers
1k views

What is momentum in neural network?

While using "Two class neural network" in Azure ML, I encountered "Momentum" property. As per documentation, which is not clear, it says For The momentum, type a value to apply ...
1
vote
0answers
41 views

How are batch gradients computed on embedding layers?

Consider the following model, which is more or less a 12-dimensional vector lookup table with 10 rows, initialized to all zeros. ...
1
vote
1answer
61 views

GAN: Discriminator converges, generator learns almost nothing

In my GAN, the discriminator loss goes down steadily, while the generator loss oscillates / does not converge. I suspect this is due to the vanishing gradient problem. Theory: as the discriminator ...
0
votes
1answer
48 views

Neural Net Backprop Weight updating Pseudo code help please

Here is my code for Backpropagation weight updating. It's a simple network with 1 hidden layer and 1 output neuron. The activation function of both hidden and output layer uses tanh. I propagate the ...
1
vote
1answer
29 views

How to decide if gradients are vanishing?

I am trying to debug a neural network. I am seeing gradients close to zero. How can I decide whether these gradients are vanishing or not? Is there some threshold to decide on vanishing gradient by ...
0
votes
1answer
847 views
0
votes
2answers
36 views

Intuitive explanation for representing gradient in higher dimensions

I do not understand how complex networks with many parameters/dimensions can be represented in a 3D space, and form a standard cost surface just like a simple network with, say, 2 parameters. For ...
2
votes
1answer
22 views

Issues with self-implemented logistic regression

I am trying to self-implement a logistic regression algorithm to do some self-learning but I am having a bit of trouble with achieving similar accuracy to the logistic regression of sklearn. Here is ...
1
vote
1answer
59 views

Why don't we find the analytical function of the cost function?

Then we could derive it and find minimum(s). e.g. in small networks the cost function has not so many variables.
0
votes
2answers
242 views

What's the proper way to do back propagation in Deep Fully Connected Neural Network for binary classification

I tried to implement a Deep fully connected neural network for binary classification using python and numpy and used Gradient Descent as optimization algorithm. ...
1
vote
0answers
23 views

Backpropagation Mathematics with Sigmoid Output Activation and Cross Entropy Loss

I am deriving a Weight update for a simple toy network with a Sigmoid Output Layer. I need some help double checking my math to make sure I did it correctly. I am using Cross-Entropy Loss as my Loss ...
1
vote
0answers
24 views

Why is the exploding/vanishing gradient problem not solved by line search?

The problem of vanishing gradients is basically that since our step size is proportional to the gradient, if the gradient is very small, it might take a long time to reach a local minimum. So why don'...
0
votes
0answers
20 views

Is it possible to calculate gradient of a filter applied on the objects (that censor them)?

I want to find an optimal censoring function, that removes objects from a given set, as to maximise the set's quality. Suppose I am given a data set consisting of N objects, each is represented by a ...
2
votes
1answer
74 views

Row-wise Jacobian with pytorch

Suppose I have $f:\mathbb{R}^{d_i}\to\mathbb{R}^{d_o}$. Let $X \in \mathbb{R}^{n \times d_i}$ and I apply $f$ to each row of $X$, obtaining $Y = f(X) \in \mathbb{R}^{n \times d_o}$. I would like to ...
0
votes
2answers
70 views

gradient descent in n dimensions

Gradient descent in $n$ dimensions. I'm learning about the downward gradient and the youtube videos and books only show a 2d curve as the slope drops to the minimum of the curve. My question is, ...
3
votes
3answers
732 views

Using a random forest, would a RandomForest performance be less if I drop the first or the last tree?

Suppose I've trained a RandomForest model with 100 trees. I then have two cases: I drop the first tree in the model. I drop the last tree in the model. Would the model performance be less in the ...
1
vote
0answers
50 views

Transposed Convolution without using Python built-in functions

Amateur here: How can we write a 2D transposed convolution (aka deconvolution) using the steepest descent method given the following restrictions: cannot use any Python built-in functions cannot ...
3
votes
1answer
743 views
2
votes
3answers
114 views

Why do we move in the negative direction of the gradient in Gradient Descent?

It is said that backpropagation, with Gradient Descent, seeks to minimize a cost function using the formula: $$ W_{new} = W_{old} - learningRate \cdot \frac{\partial E}{\partial W} $$ My question is, ...
1
vote
1answer
53 views

vanishing gradient and gradient zero

There is a well known problem vanishing gradient in BackPropagation training of ...
0
votes
0answers
14 views

How to manufacture exploding gradients in a neural networks

it might sound silly but i have this assignment problem where i have to show how exploding gradients occur in neural nets and i need to create a neural netowork which shows this phenomenon. the ...
0
votes
1answer
157 views

Why sparse features should have bigger learning rates associated? And how Adagrad achieves this? [closed]

I was learning about Adagrad optimizer. I came to know that it has a very helpful functionality which is that we can have lower learning rates for the features that are more common and greater ...
1
vote
1answer
64 views

Confusion with Notation in the Book on Deep Learning by Ian Goodfellow et al

In chapter 6.1 on 'Example: Learning XOR', the bottom of page 168 mentions: The activation function $g$ is typically chosen to be a function that is applied element-wise, with $h_i = g(x^TW_{:,i}+c_i)...
0
votes
1answer
52 views

Wouldn't it make more sense to give less importance to gradient far away in past in AdaGrad? [closed]

This is the update equation of a weight by AdaGrad: $$w_{new} = w_{old} - \frac{lr}{\sqrt{G_{}+E}}.G_{w_{old}}$$ Where $G$ is the sum of the gradients of the same weight at previous iterations, $E$ is ...
0
votes
1answer
57 views

Why are we taking the square root of the gradient in Adagrad? [closed]

This is how we update weights with Adagrad: $$w_i = w_i - \frac{lr}{\sqrt{g_i+E}}$$ where, $w_i$ is the $i^{th}$ weight, $lr$ is the learning rate, $g_i$ is the gradient of the $i^{th}$ weight at all ...
0
votes
1answer
30 views

Finding a vector that minimize the MSE of its linear combination

I have been doing a COVID-19 related project. Here is the question: N = vector of daily new infected cases D = vector of daily deaths E[D] = estimation of daily deaths N is a n-dimensional vector, n ...
1
vote
1answer
83 views

Is it valid to use numpy.gradient to find slope of line as well as slope of curve at any point?

what is the difference between slope of the line and slope of the curve? Is it valid to use numpy.gradient to find the slope of the line and slope of the curve at ...
2
votes
0answers
48 views

Maximum Entropy Policy Gradient Derivation

I am reading through the paper on Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review by Sergey Levine. I am having a difficulty in understanding this part of the ...

1
2 3 4 5
8