Questions tagged [gradient-descent]

Gradient Descent is an algorithm for finding the minimum of a function. It iteratively calculates partial derivatives (gradients) of the function and descends in steps proportional to those partial derivatives. One major application of Gradient Descent is fitting a parameterized model to a set of data: the function to be minimized is an error function for the model.

Filter by
Sorted by
Tagged with
1
vote
1answer
15 views

Is this scheme correct for logistic regression with stochastic gradient descent

I am implementing logistic regression with stochastic gradient descent, but it is not working as expected. I've tried many epochs and different learning rates $\alpha$ but the probability of belonging ...
0
votes
0answers
5 views

Cost function in ANN converges to 0.5 and the values of outputs all converge to 0

I have written a simple ANN to understand its internal structure better. However for the past few days I could not understand why it does not perform in the expected way. The way I defined COST ...
2
votes
0answers
13 views

Plotting Gradient Descent in 3d - Contour Plots

I have generated 3 parameters along with the cost function. I have the $\theta$ lists and the cost list of 100 values from the 100 iterations. I would like to plot the last 2 parameters against cost ...
0
votes
0answers
15 views

Custom training loss with custom gradients

I am trying to write a custom loss in Tensorflow v2, for simplicity let's say that I'm using Mean Squared Error loss as follows, ...
0
votes
1answer
25 views

Understanding the concept vanishing gradient and exploding gradient problem in terms of training data

I'm trying to figure out the essence of the concepts "vanishing gradient and exploding gradient problem" in terms of real-world input-output training examples instead of in terms of the properties of ...
0
votes
1answer
20 views

Why my cost function is so high?

I am trying to implement the gradient descent algorithm from scratch and use it on the Boston dataset. Here is what I have so far: ...
0
votes
0answers
12 views

Gradient Descent on Boston Dataset

I am trying to implement the gradient descent algorithm from scratch and use it on the Boston dataset. Here is what I have so far: ...
1
vote
1answer
32 views

Difference between SVM and GD/SGD?

My colleague mentioned that a data science project is using SGD classifier. So I started reading about GD/SGD and came across a nice article about Text classification using SVM and GD. In the end of ...
0
votes
1answer
20 views

Why this calculation of weight vector in linear regression is only for small dataset?

Slides from my university says, that the following way of calculating the weight vector is suitable only for small datasets. Can you please explain, why it may be suitable for small datasets? Here, X ...
1
vote
0answers
15 views

Is there a gradient descent-based optimization algorithm that works with non-linear constraints?

I have a function to optimize with ca. 200 parameters + one constraint (sum of squares of the parameters must be equal one) This problem can be solved using Lagrange Multipliers and my intuition ...
1
vote
0answers
40 views

Can we optimize heterogeneous parameters of RBF Network using Gradient Descent?

There're three parameters in the Radial Basis Function Networks (RBFN). Centers of RBFs Width of RBFs Weights of RBFs It's a fact that Weights can be easily updated using a simple Gradient Descent. ...
1
vote
0answers
9 views

Should the weights be rotated when using SciPy full convolution?

I use SciPy's single.convolve2d in "full" mode to compute gradient w.r.t to convolution layer inputs. In my current implementation, I don't rotate filters as suggested by this article because I assume ...
1
vote
1answer
17 views

GAN: Discriminator converges, generator learns almost nothing

In my GAN, the discriminator loss goes down steadily, while the generator loss oscillates / does not converge. I suspect this is due to the vanishing gradient problem. Theory: as the discriminator ...
1
vote
1answer
18 views

What is the layer above/below in a NN?

In the lecture notes of CS231n, it says (emphasis mine) ... There are three major sources of memory to keep track of: From the intermediate volume sizes: These are the raw number of ...
3
votes
1answer
26 views

Which models can handle null values?

Unfortunately trying to google or research null values in machine learning always brings up pages trying to teach you how to impute the values instead, but I'm trying to find models that can handle ...
1
vote
0answers
19 views

Neural-Networks - preferred method for training, classification v.s. regression

As a conclusion of their paper "Efficient Backprop" (http://yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf) (§10 Discussion and Conclusion), LeCun and others conlude that the preferred method for ...
5
votes
1answer
69 views

Duplicated features for gradient descent

Suppose that our data matrix X has a duplicated column, i.e, there is a duplicated feature and the matrix is not full column rank. What happpens? I guess that we can not find a unique solution ...
1
vote
1answer
36 views

A fundamental question about the gradient descent equation

So in a classic gradient descent we have W -= epsilon * (dL / dW) However, how does this make sense if we consider that L and W have some units? Wouldn't it be ...
2
votes
1answer
31 views

Interpreting Gradients and Partial Derivatives when training Neural Networks

I am trying to understand of purpose of partial differentiation in NN training by knowing how to interpret gradients and their partial derivatives. Below is my way of interpreting them so I would like ...
3
votes
1answer
51 views

What do positive and negative gradient values mean for Convolutional Neural Network?

As we have the typicall pass of the neural network we make a forawrd pass to predict classes and then we have cost function and based on that we calculate gradients. I'm wondering what are the ...
2
votes
2answers
76 views

Forward pass vs backward pass vs backpropagation

As mentioned in the question, i have some issues understanding what are the differences between those terms. From what i have understood: 1) Forward pass: compute the output of the network given the ...
4
votes
4answers
109 views

How to automatically test for the best parameters for transformed independent variable in linear model

Let's assume that I have a linear model with $k$ variables: $y = \beta_0 + \beta_1\cdot x_1 + \dots + \beta_k \cdot x_k$. Now, I want to add variable $x_{k+1}$, but, according to domain knowledge, ...
2
votes
1answer
21 views

How curvature information in second order optimization methods helps

It is said that second order optimization methods in neural networks work better than first order because they contain information about rate of change of gradient or the curvature. This information ...
2
votes
1answer
31 views

Row-wise Jacobian with pytorch

Suppose I have $f:\mathbb{R}^{d_i}\to\mathbb{R}^{d_o}$. Let $X \in \mathbb{R}^{n \times d_i}$ and I apply $f$ to each row of $X$, obtaining $Y = f(X) \in \mathbb{R}^{n \times d_o}$. I would like to ...
0
votes
1answer
29 views

Why multiply by 2 when calculating partial derivatives during backpropagation?

I'm wondering why we multiple by 2 when calculating partial derivatives. I'm referencing the 2's that I've circled below, from here. We also see this in the python implementation, ...
0
votes
0answers
16 views

Exponentiated Gradient

I am currently trying to understand exponentiated gradient from this paper. Here is an implementation of the Algorithm in Python. So my question using exponentiated-gradient algorithm we can update ...
1
vote
1answer
22 views

How should I improve my Vectorized Gradient descent linear regression model?

I wrote a vectorized Gradient descent implementation of the linear regression model. The Dataset looks something like: It's Not Working properly as I am getting negative R Squared error I don't ...
2
votes
2answers
64 views

Does Feature Normalization affect Gradient Descent | Linear Regression

am new to datascience and i want to learn linear regression so i coded linear regression from scratch and performed gradient descent to find the best $w_\theta$ and $b_\theta$ values using a tutorial. ...
2
votes
0answers
29 views

How to determine the convergence of Stochastic Gradient Descent?

While coding the batch gradient descent, it is easy to code the convergence as after each iterations the cost moves towards minimum and when the change in cost tends to approach a pre-defined number, ...
0
votes
0answers
19 views

Why does gradient descent fail training a network for predicting times table?

Cross posted I am training a feedforwardnet with gradient descent traingd as backpropagation algorithm to predict times table. ...
1
vote
0answers
27 views

How can I regularize the output of a layer from scratch (without using Keras)?

I am trying to build a Convolutional Neural Network after reading notes from Stanford's cs231n course. I use ELU activation as activation function, and SoftMax as my classifier. Architecture is simple:...
2
votes
1answer
32 views

Why in this case are gradient steps not perpendicular to contour lines?

There is a theorem that gradient at point is perpendicular to tangent line to contour line at given point. Why in this picture it seems that this rule is not respected? source: http://www....
1
vote
1answer
22 views

SGDClassifier partial_fit() for online learning - is one step of gradient descent enough?

I'm interested in incremental (online) learning for my logistic regression model trained with SGDClassifier. Basically updating the model as more labeled data comes ...
2
votes
1answer
51 views

Calculating the average of gradient decent

I am currently studying the backpropagation process and gradient decent algorithm form the book Neural Networks and Deep Learning written by Michael Nielsen and 3Blue1Brown channel in YouTube. My ...
0
votes
1answer
23 views

Cannot fig out error in my gradient function implementation in python

Im trying to implement following gradient descent function in Python for logistic regression: $∇θ(−logL)=−X^T 􏰀(y−e^{Xθ}􏰁)$ This is my python implementation: ...
0
votes
0answers
21 views

Gradient equations of gaussian kernel discriminant trained with gradiant descent

I am having a hard time trying to find the gradient equations for the weight $\alpha^t$ and $w_0$ for a gaussian kernel discriminant trained with gradient descent with the following error function $$E(...
0
votes
0answers
13 views

How is this function (for updating a Stochastic Gradient Descent model) called without a parameter?

I'm in the middle of a Deep Learning Course offered by DataCamp and the example below was given for optimizing a SGD model: As you can see, the function "get_new_model" requires one parameter: "...
0
votes
1answer
91 views

Back-propagation and stochastic gradient descent

Is backpropagation a learning method or an optimisation method? How are backpropagation and stochastic gradient descent related to each other?
3
votes
1answer
161 views

Grid search or gradient descent?

Assume we have a neural network and one if its activation functions is a function of parameter a. We want to find the weights and parameter a that leads to the minimum loss on the validation set which ...
0
votes
1answer
39 views

gradient descent for non convex function like $-x^2$

I know how to calculate gradient descent for a convex function where there is only one global minima. Also, I know methods to handle cases where the function is a non-convex function. What is really ...
1
vote
0answers
15 views

What is the difference between these 2 training scenarios?

I have a very large dataset and due to computational constraints, I have to divide the data into 20 parts (each part is around 1.5GB). I constructed a deep CNN model using Keras for this dataset. The ...
0
votes
2answers
37 views

How to form and minimise custom features for classification in supervised learning [closed]

I am having an issue in understanding how to form the features based on particular math formula, and how to adjust the weights with. The aim is to draw ellipses for each unique category of points. ...
0
votes
2answers
455 views

What is the difference between gradient descent and gradient boosting? Are they interdependent on each other by any way?

What is the difference between gradient descent and gradient boosting? Are they interdependent on each other in any way ?
0
votes
1answer
30 views

Similarity of XGBoost models?

Is xgboost with n_estimators = 100 and learning_rate = 0.1, same as xgboost with n_estimators = 50 and learning_rate = 0.2 ?
0
votes
0answers
19 views

Proof that gradient descent takes exponential time for escaping saddle points

https://arxiv.org/pdf/1705.10412.pdf I was going through this paper, and understood the crux of it. But in appendix, the complete proof of it is given which was a bit tough for me mathematically. So ...
0
votes
0answers
12 views

Purpose of gamma multiplier in gradient boosting

looking through the mathematics of gradient boosting on the relevant wikipedia page, intuitively what is the purpose of the multiplier $\gamma_i$? This term does not appear in the following ...
4
votes
1answer
38 views

How are weight updates handled in Batch Gradient Descent vs SGD?

My current understanding is that in SGD, after each data sample, the loss is used to update each weight. Ex: With 1000 samples and a network with 10 weights, there will be 10,000 individual weight ...
0
votes
0answers
17 views

Theano gradient descent and cost function issue

Sorry but can anyone point out whats wrong with this code? ...
2
votes
0answers
44 views

Linear Regression in Python using gradient descent

I am trying to implement a simple multivariate linear regression model without using any inbuilt machine libraries. So far, I have been able to get a root mean squared error for training about $2.93$ ...
0
votes
2answers
150 views

Gradient Descent or Normal Equation?

Hi guys I am really struggling with this question. I need to pick the correct choice: Suppose you have a dataset with m = 50 examples and n = 15 features for each example. You want to use ...

1 2 3 4 5 6