Questions tagged [gradient-descent]

Gradient Descent is an algorithm for finding the minimum of a function. It iteratively calculates partial derivatives (gradients) of the function and descends in steps proportional to those partial derivatives. One major application of Gradient Descent is fitting a parameterized model to a set of data: the function to be minimized is an error function for the model.

Filter by
Sorted by
Tagged with
4
votes
1answer
250 views

How to solve the gradient descent on a linear classification problem?

I have a problem which i have attached as an image. Problem is in image attached what I understand error function is given by: $e(y, \hat y)=0$ if $y \cdot a(x-b) \ge 1$ or $e(y, \hat y) = 1-y\cdot ...
0
votes
0answers
34 views

How does LSTM solve the vanishing gradient problem?

I know that there are many answers. shortly gates solve(mitigate) vanishing gradient problem. But I saw two formidable answers. Thomas Effland's answer, and Nir Abel's answer. I think they explain ...
2
votes
1answer
68 views

How many times is backprop used in epoch?

As I understand for the algorithms that use gradient descent we have to pass data to the algorithms multiple times so that the optimum is found. So one epoch means that the forward-backprop (and ...
1
vote
1answer
36 views

Is this scheme correct for logistic regression with stochastic gradient descent

I am implementing logistic regression with stochastic gradient descent, but it is not working as expected. I've tried many epochs and different learning rates $\alpha$ but the probability of belonging ...
0
votes
0answers
10 views

Cost function in ANN converges to 0.5 and the values of outputs all converge to 0

I have written a simple ANN to understand its internal structure better. However for the past few days I could not understand why it does not perform in the expected way. The way I defined COST ...
3
votes
0answers
248 views

Plotting Gradient Descent in 3d - Contour Plots

I have generated 3 parameters along with the cost function. I have the $\theta$ lists and the cost list of 100 values from the 100 iterations. I would like to plot the last 2 parameters against cost ...
0
votes
0answers
119 views

Custom training loss with custom gradients

I am trying to write a custom loss in Tensorflow v2, for simplicity let's say that I'm using Mean Squared Error loss as follows, ...
0
votes
1answer
44 views

Understanding the concept vanishing gradient and exploding gradient problem in terms of training data

I'm trying to figure out the essence of the concepts "vanishing gradient and exploding gradient problem" in terms of real-world input-output training examples instead of in terms of the properties of ...
0
votes
1answer
25 views

Why my cost function is so high?

I am trying to implement the gradient descent algorithm from scratch and use it on the Boston dataset. Here is what I have so far: ...
0
votes
0answers
111 views

Gradient Descent on Boston Dataset

I am trying to implement the gradient descent algorithm from scratch and use it on the Boston dataset. Here is what I have so far: ...
1
vote
1answer
245 views

Difference between SVM and GD/SGD?

My colleague mentioned that a data science project is using SGD classifier. So I started reading about GD/SGD and came across a nice article about Text classification using SVM and GD. In the end of ...
0
votes
1answer
29 views

Why this calculation of weight vector in linear regression is only for small dataset?

Slides from my university says, that the following way of calculating the weight vector is suitable only for small datasets. Can you please explain, why it may be suitable for small datasets? Here, X ...
1
vote
0answers
24 views

Is there a gradient descent-based optimization algorithm that works with non-linear constraints?

I have a function to optimize with ca. 200 parameters + one constraint (sum of squares of the parameters must be equal one) This problem can be solved using Lagrange Multipliers and my intuition ...
1
vote
0answers
45 views

Can we optimize heterogeneous parameters of RBF Network using Gradient Descent?

There're three parameters in the Radial Basis Function Networks (RBFN). Centers of RBFs Width of RBFs Weights of RBFs It's a fact that Weights can be easily updated using a simple Gradient Descent. ...
1
vote
0answers
18 views

Should the weights be rotated when using SciPy full convolution?

I use SciPy's single.convolve2d in "full" mode to compute gradient w.r.t to convolution layer inputs. In my current implementation, I don't rotate filters as suggested by this article because I assume ...
1
vote
1answer
61 views

GAN: Discriminator converges, generator learns almost nothing

In my GAN, the discriminator loss goes down steadily, while the generator loss oscillates / does not converge. I suspect this is due to the vanishing gradient problem. Theory: as the discriminator ...
1
vote
1answer
54 views

What is the layer above/below in a NN?

In the lecture notes of CS231n, it says (emphasis mine) ... There are three major sources of memory to keep track of: From the intermediate volume sizes: These are the raw number of ...
4
votes
3answers
528 views

Which models can handle null values?

Unfortunately trying to google or research null values in machine learning always brings up pages trying to teach you how to impute the values instead, but I'm trying to find models that can handle ...
1
vote
0answers
22 views

Neural-Networks - preferred method for training, classification v.s. regression

As a conclusion of their paper "Efficient Backprop" (http://yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf) (§10 Discussion and Conclusion), LeCun and others conlude that the preferred method for ...
6
votes
2answers
145 views

Duplicated features for gradient descent

Suppose that our data matrix X has a duplicated column, i.e, there is a duplicated feature and the matrix is not full column rank. What happpens? I guess that we can not find a unique solution ...
1
vote
1answer
39 views

A fundamental question about the gradient descent equation

So in a classic gradient descent we have W -= epsilon * (dL / dW) However, how does this make sense if we consider that L and W have some units? Wouldn't it be ...
3
votes
1answer
312 views

Interpreting Gradients and Partial Derivatives when training Neural Networks

I am trying to understand of purpose of partial differentiation in NN training by knowing how to interpret gradients and their partial derivatives. Below is my way of interpreting them so I would like ...
3
votes
1answer
310 views

What do positive and negative gradient values mean for Convolutional Neural Network?

As we have the typicall pass of the neural network we make a forawrd pass to predict classes and then we have cost function and based on that we calculate gradients. I'm wondering what are the ...
2
votes
2answers
2k views

Forward pass vs backward pass vs backpropagation

As mentioned in the question, i have some issues understanding what are the differences between those terms. From what i have understood: 1) Forward pass: compute the output of the network given the ...
4
votes
4answers
126 views

How to automatically test for the best parameters for transformed independent variable in linear model

Let's assume that I have a linear model with $k$ variables: $y = \beta_0 + \beta_1\cdot x_1 + \dots + \beta_k \cdot x_k$. Now, I want to add variable $x_{k+1}$, but, according to domain knowledge, ...
2
votes
1answer
35 views

How curvature information in second order optimization methods helps

It is said that second order optimization methods in neural networks work better than first order because they contain information about rate of change of gradient or the curvature. This information ...
2
votes
1answer
74 views

Row-wise Jacobian with pytorch

Suppose I have $f:\mathbb{R}^{d_i}\to\mathbb{R}^{d_o}$. Let $X \in \mathbb{R}^{n \times d_i}$ and I apply $f$ to each row of $X$, obtaining $Y = f(X) \in \mathbb{R}^{n \times d_o}$. I would like to ...
0
votes
1answer
37 views

Why multiply by 2 when calculating partial derivatives during backpropagation?

I'm wondering why we multiple by 2 when calculating partial derivatives. I'm referencing the 2's that I've circled below, from here. We also see this in the python implementation, ...
2
votes
0answers
48 views

Maximum Entropy Policy Gradient Derivation

I am reading through the paper on Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review by Sergey Levine. I am having a difficulty in understanding this part of the ...
0
votes
0answers
106 views

Exponentiated Gradient

I am currently trying to understand exponentiated gradient from this paper. Here is an implementation of the Algorithm in Python. So my question using exponentiated-gradient algorithm we can update ...
2
votes
1answer
36 views

How should I improve my Vectorized Gradient descent linear regression model?

I wrote a vectorized Gradient descent implementation of the linear regression model. The Dataset looks something like: It's Not Working properly as I am getting negative R Squared error I don't ...
2
votes
2answers
212 views

Does Feature Normalization affect Gradient Descent | Linear Regression

am new to datascience and i want to learn linear regression so i coded linear regression from scratch and performed gradient descent to find the best $w_\theta$ and $b_\theta$ values using a tutorial. ...
2
votes
0answers
61 views

How to determine the convergence of Stochastic Gradient Descent?

While coding the batch gradient descent, it is easy to code the convergence as after each iterations the cost moves towards minimum and when the change in cost tends to approach a pre-defined number, ...
1
vote
0answers
39 views

How can I regularize the output of a layer from scratch (without using Keras)?

I am trying to build a Convolutional Neural Network after reading notes from Stanford's cs231n course. I use ELU activation as activation function, and SoftMax as my classifier. Architecture is simple:...
2
votes
1answer
144 views

Why in this case are gradient steps not perpendicular to contour lines?

There is a theorem that gradient at point is perpendicular to tangent line to contour line at given point. Why in this picture it seems that this rule is not respected? source: http://www....
1
vote
1answer
228 views

SGDClassifier partial_fit() for online learning - is one step of gradient descent enough?

I'm interested in incremental (online) learning for my logistic regression model trained with SGDClassifier. Basically updating the model as more labeled data comes ...
2
votes
1answer
139 views

Calculating the average of gradient decent

I am currently studying the backpropagation process and gradient decent algorithm form the book Neural Networks and Deep Learning written by Michael Nielsen and 3Blue1Brown channel in YouTube. My ...
0
votes
1answer
45 views

Cannot fig out error in my gradient function implementation in python

Im trying to implement following gradient descent function in Python for logistic regression: $∇θ(−logL)=−X^T 􏰀(y−e^{Xθ}􏰁)$ This is my python implementation: ...
0
votes
0answers
13 views

How is this function (for updating a Stochastic Gradient Descent model) called without a parameter?

I'm in the middle of a Deep Learning Course offered by DataCamp and the example below was given for optimizing a SGD model: As you can see, the function "get_new_model" requires one parameter: "...
0
votes
1answer
463 views

Back-propagation and stochastic gradient descent

Is backpropagation a learning method or an optimisation method? How are backpropagation and stochastic gradient descent related to each other?
3
votes
1answer
809 views

Grid search or gradient descent?

Assume we have a neural network and one if its activation functions is a function of parameter a. We want to find the weights and parameter a that leads to the minimum loss on the validation set which ...
0
votes
1answer
68 views

gradient descent for non convex function like $-x^2$

I know how to calculate gradient descent for a convex function where there is only one global minima. Also, I know methods to handle cases where the function is a non-convex function. What is really ...
1
vote
0answers
18 views

What is the difference between these 2 training scenarios?

I have a very large dataset and due to computational constraints, I have to divide the data into 20 parts (each part is around 1.5GB). I constructed a deep CNN model using Keras for this dataset. The ...
0
votes
2answers
38 views

How to form and minimise custom features for classification in supervised learning [closed]

I am having an issue in understanding how to form the features based on particular math formula, and how to adjust the weights with. The aim is to draw ellipses for each unique category of points. ...
1
vote
2answers
2k views

What is the difference between gradient descent and gradient boosting? Are they interdependent on each other by any way?

What is the difference between gradient descent and gradient boosting? Are they interdependent on each other in any way ?
0
votes
1answer
49 views

Similarity of XGBoost models?

Is xgboost with n_estimators = 100 and learning_rate = 0.1, same as xgboost with n_estimators = 50 and learning_rate = 0.2 ?
1
vote
0answers
26 views

Purpose of gamma multiplier in gradient boosting

looking through the mathematics of gradient boosting on the relevant wikipedia page, intuitively what is the purpose of the multiplier $\gamma_i$? This term does not appear in the following ...
4
votes
1answer
111 views

How are weight updates handled in Batch Gradient Descent vs SGD?

My current understanding is that in SGD, after each data sample, the loss is used to update each weight. Ex: With 1000 samples and a network with 10 weights, there will be 10,000 individual weight ...
2
votes
1answer
110 views

Linear Regression in Python using gradient descent

I am trying to implement a simple multivariate linear regression model without using any inbuilt machine libraries. So far, I have been able to get a root mean squared error for training about $2.93$ ...
1
vote
2answers
3k views

Gradient Descent or Normal Equation?

Suppose you have a dataset with m = 50 examples and n = 15 features for each example. You want to use multivariate linear regression to fit the parameters theta to our data. Should you prefer gradient ...

1 2
3
4 5
8