Questions tagged [gradient-descent]

Gradient Descent is an algorithm for finding the minimum of a function. It iteratively calculates partial derivatives (gradients) of the function and descends in steps proportional to those partial derivatives. One major application of Gradient Descent is fitting a parameterized model to a set of data: the function to be minimized is an error function for the model.

Filter by
Sorted by
Tagged with
0
votes
0answers
5 views

When to use Gradient boosting over stochastic gradient boosting

Gradient boosting works on the Gradient Descent concept and it's one of the ensemble methods. It has a regularization parameter to select subsamples, which is called stochastic gradient boosting. ...
2
votes
1answer
27 views

Adam Optimiser First Step

Plotting the paths on the cost surface from different gradient descent optimisers on a toy example, I found that the Adam algorithm does not initially travel in the direction of steepest gradient (...
1
vote
1answer
26 views

Derivative of a custom loss function with the logistic function

I have costum loss function with $\mu ,p, o, u, v$ as variables and $\sigma$ is the logistic function. I need to derive this loss function. Due to multiple variables in the loss function, I need to ...
1
vote
0answers
13 views

Can we talk about vanishing activations?

When updating the weights of a deep neural network using backpropagation, to update the weights of a given hidden layer, we use both the partial derivatives of the objective function with respect to ...
0
votes
1answer
14 views

Why are mini-batches degrading my conv net MNIST classifier?

I've made a conv net from scratch in python to classify the MNIST handwritten digits (centralized). It's composed of a single convolutional network with 8 3x3 kernels, a 2x2 maxpool layer and a 10 ...
0
votes
0answers
11 views

When should I update weights and biases in Neural Network?

So, I am building a Neural Network from scratch for (typically) classifying MNIST digits. Everything is going fine, I can get up to 85% accuracy accross all testing data with stochastic gradient ...
0
votes
0answers
17 views

Unbiased Predictions for all Distinct Training Subsets

Suppose I have a data set $\left(X_i \in \chi, y_i \in \zeta \right)$ where $X_i$ and $y_i$ correspond to instances and labels, and $\chi$ and $\zeta$ correspond to the space where $X_i$ and $y_i$ ...
6
votes
1answer
144 views

Changing the batch size during training

The choice of batch size is in some sense the measure of stochasticity : On one hand, smaller batch sizes make the gradient descent more stochastic, the SGD can deviate significantly from the exact ...
2
votes
3answers
608 views

Why a sign of gradient (plus or minus) is not enough for finding a steepest ascend?

Consider a simple 1-D function $y = x^2$ to find a maximum with the gradient ascent method. If we start in point 3 on x-axis: $$ \frac{\partial f}{\partial x} \...
1
vote
0answers
16 views

Dying gradient issue in Graph Neural Networks

I am using Pytorch-Geometric library to implement a Graph Convolutional Layer(GCN) followed by few linear layers for a prediction task. But after training on graphs with np. of nodes being 10K and no. ...
0
votes
1answer
18 views

Neural Network Optimization steps order

I have a very basic question on the optimization algotithm, when I'm adjusting weights and biases in a NN, should I: Forward propagate and backpropagate to calculate gradient descent (DC) for each ...
1
vote
0answers
40 views

OLS and gradient descent difference?

I am doing a course on Udemy on which the instructor applied OLS (Ordinary least square) on a housing dataset. The curve he got was linear,with parameters [10^5,239]. Now When I tried to repeat the ...
3
votes
1answer
30 views

what is difference between Logistic regression and SGDClassifier with log loss OR SVM and SGDClassifer with hinge loss?

Can we just use SGDClassifier with log loss instead of Logistic regression, would they have similar results ?
1
vote
1answer
22 views

Do zero weights receive zero gradient in ReLU neural networks?

Suppose I have a deep neural network using the ReLU activation function, that is $\sigma(x) = max(x, 0)$. Suppose some weight $w_i$ becomes exactly $0$ at some point. Am I getting something wrong here,...
1
vote
0answers
6 views

Error term in probabilistic interpretation of least squares update rule

I have read in Stanford's CS229 course notes that to justify the least-squares update rule with probability, the following is assumed: $$y^{(i)} = \theta^Tx^{(i)}+\epsilon^{(i)}$$ , where $\epsilon^{(...
0
votes
0answers
21 views

Why is this equation converted to matrix form in this way? Is it possible to multiply an inverse matrix with a vector?

I have been banging my head on wall for days trying to decode this equation. please help me out with this... Below is the equation (consider $x$ as $\Delta x$, and $y$ as $\Delta y$): $x = - \eta(Id-\...
0
votes
0answers
12 views

How to interpret gradient descent in boosting ensembles?

I struggle to grasp the role of gradient based optimization in boosting ensembles. As far as I understand boosting means combining a bunch of estimators (of the same types, usually decision trees) ...
0
votes
0answers
31 views

Multivariable linear gradient descent resulting in inf

I am trying to implement a multivariable gradient descent algorithm, it seems to start working fine, and works on smaller datasets, but applying it to larger datasets the variables overflow and cause ...
-1
votes
3answers
42 views

Gradient descent does not converge in some runs and converges in other runs in the following simple Keras network

When training a simple Keras NN (1 input, 1 level with 1 unit for a regression task) during some runs I get big constant loss that does not change in 80 batches. During other runs it decreases. What ...
0
votes
2answers
40 views

Why use gradient descent on Deep Nets / RNNs when cost function is not convex?

Why do we use gradient descent on very non-convex loss functions such as in Deep nets / RNNs rather than a heuristic search (genetic algorithms, simulated annealing, etc)?
5
votes
2answers
352 views

Getting NN weights for every batch / epoch from Keras model

I am trying to get weights for every batch / epoch from Keras model after it is trained. To do so I use callback to make model save weights during training. Yet after model is trained it looks like I ...
1
vote
0answers
29 views

SVM with gradient descent

The constrained optimization problem in SVM is given by min 1/2 ||w||^2 s.t y(i)(w^T x(i) + b >= 1 for all i Now converting this to an unconstrained optimization problem gives the lagriangian L as ...
3
votes
2answers
167 views

How does the descending gradient know what weights to adjust?

I was reading about descending gradient. How does the descending gradient know what weights to adjust? Does it adjust to all network weights at the same time? Does each weight have an associated error?...
2
votes
3answers
69 views

Neural Network Loss Function - Mean Square Error: questions about what 'n' signifies

I'm very new to neural networks and have recently learnt about the loss functions used with neural networks. This question is in regards to the mean square error metric, defined as (from the textbook ...
1
vote
1answer
20 views

Maximum Likelihood with Gradient Descent or Coordinate Descent blows up

Context The maximum likelihood estimators for a Normal distribution with unknown mean and unknown variance are $$ \widehat{\mu} = \frac{1}{n}\sum_{i=1}^n x_i \qquad \text{and} \qquad \widehat{\sigma}^...
1
vote
1answer
42 views

Why standard distribution for ML [closed]

Data normalization: It ensures that each input (each pixel value, in this case) comes from a standard distribution. This standardization makes our model train and reach a minimum error, faster! my ...
3
votes
0answers
81 views

Why the sigmoid activation function results in sub-optimal gradient descent?

I need some help understanding the second shortcoming of the sigmoid activation function as described in this video from Stanford. She says that because the output of sigmoid is always positive, that ...
1
vote
0answers
58 views

How are batch gradients computed on embedding layers?

Consider the following model, which is more or less a 12-dimensional vector lookup table with 10 rows, initialized to all zeros. ...
1
vote
1answer
31 views

How to decide if gradients are vanishing?

I am trying to debug a neural network. I am seeing gradients close to zero. How can I decide whether these gradients are vanishing or not? Is there some threshold to decide on vanishing gradient by ...
5
votes
2answers
3k views

What is momentum in neural network?

While using "Two class neural network" in Azure ML, I encountered "Momentum" property. As per documentation, which is not clear, it says For The momentum, type a value to apply ...
1
vote
1answer
61 views

Why don't we find the analytical function of the cost function?

Then we could derive it and find minimum(s). e.g. in small networks the cost function has not so many variables.
1
vote
0answers
49 views

Backpropagation Mathematics with Sigmoid Output Activation and Cross Entropy Loss

I am deriving a Weight update for a simple toy network with a Sigmoid Output Layer. I need some help double checking my math to make sure I did it correctly. I am using Cross-Entropy Loss as my Loss ...
1
vote
1answer
30 views

Why is the exploding/vanishing gradient problem not solved by line search?

The problem of vanishing gradients is basically that since our step size is proportional to the gradient, if the gradient is very small, it might take a long time to reach a local minimum. So why don'...
0
votes
0answers
20 views

Is it possible to calculate gradient of a filter applied on the objects (that censor them)?

I want to find an optimal censoring function, that removes objects from a given set, as to maximise the set's quality. Suppose I am given a data set consisting of N objects, each is represented by a ...
3
votes
3answers
751 views

Using a random forest, would a RandomForest performance be less if I drop the first or the last tree?

Suppose I've trained a RandomForest model with 100 trees. I then have two cases: I drop the first tree in the model. I drop the last tree in the model. Would the model performance be less in the ...
1
vote
0answers
15 views

How to manufacture exploding gradients in a neural networks

it might sound silly but i have this assignment problem where i have to show how exploding gradients occur in neural nets and i need to create a neural netowork which shows this phenomenon. the ...
2
votes
3answers
188 views

Why do we move in the negative direction of the gradient in Gradient Descent?

It is said that backpropagation, with Gradient Descent, seeks to minimize a cost function using the formula: $$ W_{new} = W_{old} - learningRate \cdot \frac{\partial E}{\partial W} $$ My question is, ...
1
vote
1answer
74 views

vanishing gradient and gradient zero

There is a well known problem vanishing gradient in BackPropagation training of ...
0
votes
1answer
218 views

Why sparse features should have bigger learning rates associated? And how Adagrad achieves this? [closed]

I was learning about Adagrad optimizer. I came to know that it has a very helpful functionality which is that we can have lower learning rates for the features that are more common and greater ...
1
vote
1answer
69 views

Confusion with Notation in the Book on Deep Learning by Ian Goodfellow et al

In chapter 6.1 on 'Example: Learning XOR', the bottom of page 168 mentions: The activation function $g$ is typically chosen to be a function that is applied element-wise, with $h_i = g(x^TW_{:,i}+c_i)...
0
votes
1answer
57 views

Wouldn't it make more sense to give less importance to gradient far away in past in AdaGrad? [closed]

This is the update equation of a weight by AdaGrad: $$w_{new} = w_{old} - \frac{lr}{\sqrt{G_{}+E}}.G_{w_{old}}$$ Where $G$ is the sum of the gradients of the same weight at previous iterations, $E$ is ...
0
votes
1answer
98 views

Why are we taking the square root of the gradient in Adagrad? [closed]

This is how we update weights with Adagrad: $$w_i = w_i - \frac{lr}{\sqrt{g_i+E}}$$ where, $w_i$ is the $i^{th}$ weight, $lr$ is the learning rate, $g_i$ is the gradient of the $i^{th}$ weight at all ...
2
votes
1answer
30 views

Issues with self-implemented logistic regression

I am trying to self-implement a logistic regression algorithm to do some self-learning but I am having a bit of trouble with achieving similar accuracy to the logistic regression of sklearn. Here is ...
0
votes
2answers
95 views

Why Mini batch gradient descent is faster than gradient descent?

According to me: Mini Batch Gradient Descent : 1.It takes a specified batch number say 32. 2.Evaluate loss on 32 examples. 3.Update weights. 4.Repeat until every example is complete. 5.Repeat till a ...
0
votes
1answer
48 views

With Stochastic Gradient Descent why we dont compute exact derivative of loss function?

In a blog I read this: With Stochastic Gradient Descent we don’t compute the exact derivate of our loss function. Instead, we’re estimating it on a small batch. blog. Now I am confused with the whole ...
0
votes
0answers
40 views

Why do we use stochastic gradient descent in neural networks and what are the main ideas behind this optimization technique?

I am a student and I am studying machine learning. I am focusing on neural network, and I have seen that for a neural network, to define the optimal weights, we don't use gradient descent, but we use ...
1
vote
1answer
49 views

Why do we only care about convex functions when doing Gradient Descent/SGD?

I mean I know why we specifically care about convex functions: it's because their local minimum are also global, and so you just have to "follow a path which goes down" to find the minima of ...
0
votes
1answer
57 views

Linear function gradient descent

I am trying to implement a gradient descent algorithm for a simple linear function: y(x) = x Where initial hypothesis function is: ...
3
votes
1answer
53 views

When does it make sense to choose gradient descent for SVM over liblinear?

I understand using gradient descent methods with SVM is intractable if you've used the kernel trick. In that case, best to use libsvm as your solver. But in the case that you are not using a kernel ...

1
2 3 4 5
8