Questions tagged [gradient-descent]

Gradient Descent is an algorithm for finding the minimum of a function. It iteratively calculates partial derivatives (gradients) of the function and descends in steps proportional to those partial derivatives. One major application of Gradient Descent is fitting a parameterized model to a set of data: the function to be minimized is an error function for the model.

Filter by
Sorted by
Tagged with
0
votes
0answers
5 views

Does severe multicollinearity affect solving linear regression by gradient descent?

Since OLS may fail when there is severe/near perfect multicollinearity, how would gradient descent perform in such a scenario? Does it converge at the minima? (My guess is, Cost function of linear ...
0
votes
0answers
6 views

Tensorflow - Manually decay Adam optimizer

I've been experimenting with reinforcement learning and using the train_on_batch method of tf.keras.models.Model to update the ...
0
votes
0answers
6 views

Projected gradient descent in keras

I am currently working on a project and I need to do project gradient descent instead of vanilla gradient descent on a network. I am unsure if current deep learning frameworks have that functionality. ...
1
vote
0answers
40 views

Gradient boosting - Learning rate decay

Learning rate decay is a very used technique for training neural networks. Common beliefs are that: an initially large learning rate accelerates training or helps the network escape local minima; ...
3
votes
2answers
208 views

What is the best way to find minima in Logistic regression?

In the Andrew NG's tutorial on Machine Learning, he takes the first derivative of the error function then takes small steps in direction of the derivative to find minima. (Gradient Descent basically) ...
0
votes
1answer
39 views

Should bias updates be porportional to overfitting?

According to questions on the internet, the bias is a learnable parameter, and there are different solutions to updating it, but I failed to find a concise methodology of correctly updating biases ...
1
vote
1answer
13 views

line search in gradient descent applied to a convex function

I have been working on implementing a line search method for gradient descent where I made an assumption that at any given point on my surface of the loss function I can reach the minima by the single ...
4
votes
1answer
160 views

How to solve the gradient descent on a linear classification problem?

I have a problem which i have attached as an image. Problem is in image attached what I understand error function is given by: $e(y, \hat y)=0$ if $y \cdot a(x-b) \ge 1$ or $e(y, \hat y) = 1-y\cdot ...
0
votes
0answers
23 views

How does LSTM solve the vanishing gradient problem?

I know that there are many answers. shortly gates solve(mitigate) vanishing gradient problem. But I saw two formidable answers. Thomas Effland's answer, and Nir Abel's answer. I think they explain ...
2
votes
1answer
36 views

How many times is backprop used in epoch?

As I understand for the algorithms that use gradient descent we have to pass data to the algorithms multiple times so that the optimum is found. So one epoch means that the forward-backprop (and ...
1
vote
1answer
24 views

Is this scheme correct for logistic regression with stochastic gradient descent

I am implementing logistic regression with stochastic gradient descent, but it is not working as expected. I've tried many epochs and different learning rates $\alpha$ but the probability of belonging ...
0
votes
0answers
5 views

Cost function in ANN converges to 0.5 and the values of outputs all converge to 0

I have written a simple ANN to understand its internal structure better. However for the past few days I could not understand why it does not perform in the expected way. The way I defined COST ...
2
votes
0answers
29 views

Plotting Gradient Descent in 3d - Contour Plots

I have generated 3 parameters along with the cost function. I have the $\theta$ lists and the cost list of 100 values from the 100 iterations. I would like to plot the last 2 parameters against cost ...
0
votes
0answers
24 views

Custom training loss with custom gradients

I am trying to write a custom loss in Tensorflow v2, for simplicity let's say that I'm using Mean Squared Error loss as follows, ...
0
votes
1answer
28 views

Understanding the concept vanishing gradient and exploding gradient problem in terms of training data

I'm trying to figure out the essence of the concepts "vanishing gradient and exploding gradient problem" in terms of real-world input-output training examples instead of in terms of the properties of ...
0
votes
1answer
20 views

Why my cost function is so high?

I am trying to implement the gradient descent algorithm from scratch and use it on the Boston dataset. Here is what I have so far: ...
0
votes
0answers
20 views

Gradient Descent on Boston Dataset

I am trying to implement the gradient descent algorithm from scratch and use it on the Boston dataset. Here is what I have so far: ...
1
vote
1answer
47 views

Difference between SVM and GD/SGD?

My colleague mentioned that a data science project is using SGD classifier. So I started reading about GD/SGD and came across a nice article about Text classification using SVM and GD. In the end of ...
0
votes
1answer
23 views

Why this calculation of weight vector in linear regression is only for small dataset?

Slides from my university says, that the following way of calculating the weight vector is suitable only for small datasets. Can you please explain, why it may be suitable for small datasets? Here, X ...
1
vote
0answers
18 views

Is there a gradient descent-based optimization algorithm that works with non-linear constraints?

I have a function to optimize with ca. 200 parameters + one constraint (sum of squares of the parameters must be equal one) This problem can be solved using Lagrange Multipliers and my intuition ...
1
vote
0answers
42 views

Can we optimize heterogeneous parameters of RBF Network using Gradient Descent?

There're three parameters in the Radial Basis Function Networks (RBFN). Centers of RBFs Width of RBFs Weights of RBFs It's a fact that Weights can be easily updated using a simple Gradient Descent. ...
1
vote
0answers
10 views

Should the weights be rotated when using SciPy full convolution?

I use SciPy's single.convolve2d in "full" mode to compute gradient w.r.t to convolution layer inputs. In my current implementation, I don't rotate filters as suggested by this article because I assume ...
1
vote
1answer
29 views

GAN: Discriminator converges, generator learns almost nothing

In my GAN, the discriminator loss goes down steadily, while the generator loss oscillates / does not converge. I suspect this is due to the vanishing gradient problem. Theory: as the discriminator ...
1
vote
1answer
22 views

What is the layer above/below in a NN?

In the lecture notes of CS231n, it says (emphasis mine) ... There are three major sources of memory to keep track of: From the intermediate volume sizes: These are the raw number of ...
3
votes
1answer
59 views

Which models can handle null values?

Unfortunately trying to google or research null values in machine learning always brings up pages trying to teach you how to impute the values instead, but I'm trying to find models that can handle ...
1
vote
0answers
20 views

Neural-Networks - preferred method for training, classification v.s. regression

As a conclusion of their paper "Efficient Backprop" (http://yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf) (§10 Discussion and Conclusion), LeCun and others conlude that the preferred method for ...
5
votes
1answer
71 views

Duplicated features for gradient descent

Suppose that our data matrix X has a duplicated column, i.e, there is a duplicated feature and the matrix is not full column rank. What happpens? I guess that we can not find a unique solution ...
1
vote
1answer
36 views

A fundamental question about the gradient descent equation

So in a classic gradient descent we have W -= epsilon * (dL / dW) However, how does this make sense if we consider that L and W have some units? Wouldn't it be ...
2
votes
1answer
73 views

Interpreting Gradients and Partial Derivatives when training Neural Networks

I am trying to understand of purpose of partial differentiation in NN training by knowing how to interpret gradients and their partial derivatives. Below is my way of interpreting them so I would like ...
3
votes
1answer
66 views

What do positive and negative gradient values mean for Convolutional Neural Network?

As we have the typicall pass of the neural network we make a forawrd pass to predict classes and then we have cost function and based on that we calculate gradients. I'm wondering what are the ...
2
votes
2answers
290 views

Forward pass vs backward pass vs backpropagation

As mentioned in the question, i have some issues understanding what are the differences between those terms. From what i have understood: 1) Forward pass: compute the output of the network given the ...
4
votes
4answers
112 views

How to automatically test for the best parameters for transformed independent variable in linear model

Let's assume that I have a linear model with $k$ variables: $y = \beta_0 + \beta_1\cdot x_1 + \dots + \beta_k \cdot x_k$. Now, I want to add variable $x_{k+1}$, but, according to domain knowledge, ...
2
votes
1answer
22 views

How curvature information in second order optimization methods helps

It is said that second order optimization methods in neural networks work better than first order because they contain information about rate of change of gradient or the curvature. This information ...
2
votes
1answer
34 views

Row-wise Jacobian with pytorch

Suppose I have $f:\mathbb{R}^{d_i}\to\mathbb{R}^{d_o}$. Let $X \in \mathbb{R}^{n \times d_i}$ and I apply $f$ to each row of $X$, obtaining $Y = f(X) \in \mathbb{R}^{n \times d_o}$. I would like to ...
0
votes
1answer
31 views

Why multiply by 2 when calculating partial derivatives during backpropagation?

I'm wondering why we multiple by 2 when calculating partial derivatives. I'm referencing the 2's that I've circled below, from here. We also see this in the python implementation, ...
0
votes
0answers
21 views

Exponentiated Gradient

I am currently trying to understand exponentiated gradient from this paper. Here is an implementation of the Algorithm in Python. So my question using exponentiated-gradient algorithm we can update ...
1
vote
1answer
23 views

How should I improve my Vectorized Gradient descent linear regression model?

I wrote a vectorized Gradient descent implementation of the linear regression model. The Dataset looks something like: It's Not Working properly as I am getting negative R Squared error I don't ...
2
votes
2answers
74 views

Does Feature Normalization affect Gradient Descent | Linear Regression

am new to datascience and i want to learn linear regression so i coded linear regression from scratch and performed gradient descent to find the best $w_\theta$ and $b_\theta$ values using a tutorial. ...
2
votes
0answers
32 views

How to determine the convergence of Stochastic Gradient Descent?

While coding the batch gradient descent, it is easy to code the convergence as after each iterations the cost moves towards minimum and when the change in cost tends to approach a pre-defined number, ...
0
votes
0answers
20 views

Why does gradient descent fail training a network for predicting times table?

Cross posted I am training a feedforwardnet with gradient descent traingd as backpropagation algorithm to predict times table. ...
1
vote
0answers
28 views

How can I regularize the output of a layer from scratch (without using Keras)?

I am trying to build a Convolutional Neural Network after reading notes from Stanford's cs231n course. I use ELU activation as activation function, and SoftMax as my classifier. Architecture is simple:...
2
votes
1answer
44 views

Why in this case are gradient steps not perpendicular to contour lines?

There is a theorem that gradient at point is perpendicular to tangent line to contour line at given point. Why in this picture it seems that this rule is not respected? source: http://www....
1
vote
1answer
49 views

SGDClassifier partial_fit() for online learning - is one step of gradient descent enough?

I'm interested in incremental (online) learning for my logistic regression model trained with SGDClassifier. Basically updating the model as more labeled data comes ...
2
votes
1answer
56 views

Calculating the average of gradient decent

I am currently studying the backpropagation process and gradient decent algorithm form the book Neural Networks and Deep Learning written by Michael Nielsen and 3Blue1Brown channel in YouTube. My ...
0
votes
1answer
27 views

Cannot fig out error in my gradient function implementation in python

Im trying to implement following gradient descent function in Python for logistic regression: $∇θ(−logL)=−X^T 􏰀(y−e^{Xθ}􏰁)$ This is my python implementation: ...
0
votes
0answers
23 views

Gradient equations of gaussian kernel discriminant trained with gradiant descent

I am having a hard time trying to find the gradient equations for the weight $\alpha^t$ and $w_0$ for a gaussian kernel discriminant trained with gradient descent with the following error function $$E(...
0
votes
0answers
13 views

How is this function (for updating a Stochastic Gradient Descent model) called without a parameter?

I'm in the middle of a Deep Learning Course offered by DataCamp and the example below was given for optimizing a SGD model: As you can see, the function "get_new_model" requires one parameter: "...
0
votes
1answer
136 views

Back-propagation and stochastic gradient descent

Is backpropagation a learning method or an optimisation method? How are backpropagation and stochastic gradient descent related to each other?
3
votes
1answer
243 views

Grid search or gradient descent?

Assume we have a neural network and one if its activation functions is a function of parameter a. We want to find the weights and parameter a that leads to the minimum loss on the validation set which ...
0
votes
1answer
47 views

gradient descent for non convex function like $-x^2$

I know how to calculate gradient descent for a convex function where there is only one global minima. Also, I know methods to handle cases where the function is a non-convex function. What is really ...

1
2 3 4 5 6