Questions tagged [gradient-descent]

Gradient Descent is an algorithm for finding the minimum of a function. It iteratively calculates partial derivatives (gradients) of the function and descends in steps proportional to those partial derivatives. One major application of Gradient Descent is fitting a parameterized model to a set of data: the function to be minimized is an error function for the model.

109 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
5
votes
0answers
472 views

differences between LSQR and FTRL when working with very sparse data

I have a 2M instances dataset with millions of very very sparse dummy variables created using the hashing trick = ...
4
votes
1answer
79 views

Learning parameters when loss is a piecewise function

I have a network to generate a single number $T$. I know in advance: a property of the loss function is that, when $T \in [a_1, a_2]$, the loss has the same value $L_1$; when $T \in [a_2, a_3]$, the ...
3
votes
0answers
70 views

Why the sigmoid activation function results in sub-optimal gradient descent?

I need some help understanding the second shortcoming of the sigmoid activation function as described in this video from Stanford. She says that because the output of sigmoid is always positive, that ...
3
votes
0answers
248 views

Plotting Gradient Descent in 3d - Contour Plots

I have generated 3 parameters along with the cost function. I have the $\theta$ lists and the cost list of 100 values from the 100 iterations. I would like to plot the last 2 parameters against cost ...
3
votes
1answer
251 views

Fast Python implementation of the gradient descent

I'm looking for fast Python implémentations of gradient descent optimization algorithm. I have a convex problem , with no constraint, so for now I'm using the BFGS algorithm implemented in scikit-...
2
votes
1answer
22 views

Issues with self-implemented logistic regression

I am trying to self-implement a logistic regression algorithm to do some self-learning but I am having a bit of trouble with achieving similar accuracy to the logistic regression of sklearn. Here is ...
2
votes
0answers
183 views

Why L2 norm in AdaGrad update equation not L1?

The update equation of AdaGrad is as follows: I understand that sparse features have small updates and this is a problem. I understand that the idea of AdaGrad is to make the update speed (learning ...
2
votes
0answers
45 views

Dissecting and understanding the Adam optimization's formula

Adam's optimization has the fololwing parameter update rule : $$ \theta_{t+1} = \theta_{t} - \alpha*\dfrac{m_t}{\sqrt{v_t + \epsilon}}$$ where $$ m_t \text{ is first moment of gradients and} \space ...
2
votes
1answer
74 views

Row-wise Jacobian with pytorch

Suppose I have $f:\mathbb{R}^{d_i}\to\mathbb{R}^{d_o}$. Let $X \in \mathbb{R}^{n \times d_i}$ and I apply $f$ to each row of $X$, obtaining $Y = f(X) \in \mathbb{R}^{n \times d_o}$. I would like to ...
2
votes
0answers
48 views

Maximum Entropy Policy Gradient Derivation

I am reading through the paper on Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review by Sergey Levine. I am having a difficulty in understanding this part of the ...
2
votes
0answers
61 views

How to determine the convergence of Stochastic Gradient Descent?

While coding the batch gradient descent, it is easy to code the convergence as after each iterations the cost moves towards minimum and when the change in cost tends to approach a pre-defined number, ...
2
votes
1answer
388 views

Intractability in Variational Autoencoders

I'm having difficulty understanding when integrals are intractable in variational inference problems. In a variational autoencoder with observation $x$ and latent variable $z$ we want to maximize data ...
2
votes
0answers
27 views

Get derivatives from your NN

How can I get the gradient of a node in the NN with respect to another one? I need to train a NN, which for the sake of simplicity has 2 neurons as input (x, y), a neuron as a bottleneck (z), and 2 ...
2
votes
1answer
819 views

multilayer perceptron do not converge

I have been coding my own multi layer perceptron in MATLAB and it can be compiled without error. My training data features,x, has values from 1 to 360, and training data output, y, has the value of ...
2
votes
0answers
181 views

Explanations about ADAM Optimizer algorithm

I'm a beginner in Machine learning and i'm searching for some optimizer for the gradient descent. I've searched many topics about that, and did a state of art of all these optimizers. I have just one ...
2
votes
0answers
156 views

Vowpal Wabbit Online Normalization — Possible to parallelize?

Vowpal Wabbit (VW) uses online normalization as explained here [1]. When running VW with multiple workers, workers synchronize their models with an AllReduce at the end of each epoch. Is it possible ...
2
votes
0answers
146 views

SGD learning gets stuck when using a max pooling layer (but it works fine with just conv + fc)

I'm working on a CNN library for a university project and I'm having some trouble implementing the backpropagation through the max pooling layer. Please note that the whole thing was built from ...
2
votes
0answers
95 views

What are some machine learning problems that can be attacked with continuous multiobjective optimization?

I am working on continuous vector optimization, and hence continuous multiobjective optimization is a particular case. I am interested in finding applications in machine learning for this problems. Is ...
2
votes
0answers
124 views

Why is each successive tree in GBM fit on the negative gradient of the loss function?

Page 359 of Elements Of Statistical Learning 2nd edition says the below. Can someone explain the intuition & simplify it in layman terms? Questions What is the reason/intuition & math ...
2
votes
0answers
203 views

Non-linear data preprocessing before mini-batch gradient descent

I found some realisation of pretty interesting ML algorithm. https://github.com/EderSantana/DeepEEG We have dataset X, matrix of spacial filter W, matrix of time filter V, and matrices U and B - ...
2
votes
0answers
76 views

Regression problem - too complex for gradient descent

I try to predict temperatures values as function of time and different parameters. The temperature curve look like a "ramp" with some "gauss peaks" on regular intervals. So, I try to build a ...
2
votes
0answers
285 views

How do I deal with non-IID data in gradient boosted random forest (for stock market)?

I am working on a stock market decision system. I have currently centered on gradient boosting as the likely best machine learning solution for the problem. However, I have 2 fundamental issues with ...
1
vote
0answers
29 views

SVM with gradient descent

The constrained optimization problem in SVM is given by min 1/2 ||w||^2 s.t y(i)(w^T x(i) + b >= 1 for all i Now converting this to an unconstrained optimization problem gives the lagriangian L as ...
1
vote
0answers
41 views

How are batch gradients computed on embedding layers?

Consider the following model, which is more or less a 12-dimensional vector lookup table with 10 rows, initialized to all zeros. ...
1
vote
0answers
23 views

Backpropagation Mathematics with Sigmoid Output Activation and Cross Entropy Loss

I am deriving a Weight update for a simple toy network with a Sigmoid Output Layer. I need some help double checking my math to make sure I did it correctly. I am using Cross-Entropy Loss as my Loss ...
1
vote
0answers
24 views

Why is the exploding/vanishing gradient problem not solved by line search?

The problem of vanishing gradients is basically that since our step size is proportional to the gradient, if the gradient is very small, it might take a long time to reach a local minimum. So why don'...
1
vote
1answer
42 views

Why do we only care about convex functions when doing Gradient Descent/SGD?

I mean I know why we specifically care about convex functions: it's because their local minimum are also global, and so you just have to "follow a path which goes down" to find the minima of ...
1
vote
1answer
68 views

Do batch GD and stochastic GD give the same results?

If a neural network is trained on a dataset of M samples for N epochs, do batch GD and SGD give the same result? Is SGD is faster because utilize the hardware better? I am asking because I figured out ...
1
vote
1answer
51 views

How to find slope of curve at certain points

how to find slope at certain points circled in blue in below curve ? Are these below 2 approaches valid ? though they give different results . How to automatically find the points where the slope ...
1
vote
1answer
57 views

Why does Siamese neural networks use tied weights and how do they work?

Reading this paper on one-shot learning "Siamese Neural Networks for One-shot Image Recognition" I was introduced to the idea of Siamese Neural Networks. What I did not fully grasp was what they ...
1
vote
0answers
56 views

Derivative of Loss wrt bias term

I read this and have an ambiguity. I try to understand well how to calculate the derivative of Loss w.r.t to bias. In this question, we have this definition: ...
1
vote
0answers
18 views

Tuning parameters for gradient boosting/xgboost

In practice, which parameter do you typically tune first? Do you tune the learning rate (or step size) first? and then tune the total number of iterations? And how do you go about tuning these ...
1
vote
0answers
15 views

Matlab Optimization. Meaning of warning: “The slope should be 2. It appears to be 1.”

I'm using the manopt package to solve some optimization problems in matlab. The problem is of the form. problem.cost = @(x) f(x) problem.egrad = @(x) g(x) After the problem definition, I check ...
1
vote
0answers
15 views

Gradient calculation for proportional odds (logistic) model

I am trying to calculate a gradient for a proportional odds model. http://fa.bianp.net/blog/2013/logistic-ordinal-regression/ What steps are required to take the derivative with respect to w? $$\...
1
vote
1answer
68 views

gradient descent diverges extremely

I have manually created a random data set around some mean value and I have tried to use gradient descent linear regression to predict this simple mean value. I have done exactly like in the manual ...
1
vote
0answers
50 views

Transposed Convolution without using Python built-in functions

Amateur here: How can we write a 2D transposed convolution (aka deconvolution) using the steepest descent method given the following restrictions: cannot use any Python built-in functions cannot ...
1
vote
0answers
188 views

grad-cam implementation on mobilenet SSD network

Below is a gradcam implementation for a standard image classifier : ...
1
vote
0answers
24 views

Is there a gradient descent-based optimization algorithm that works with non-linear constraints?

I have a function to optimize with ca. 200 parameters + one constraint (sum of squares of the parameters must be equal one) This problem can be solved using Lagrange Multipliers and my intuition ...
1
vote
0answers
45 views

Can we optimize heterogeneous parameters of RBF Network using Gradient Descent?

There're three parameters in the Radial Basis Function Networks (RBFN). Centers of RBFs Width of RBFs Weights of RBFs It's a fact that Weights can be easily updated using a simple Gradient Descent. ...
1
vote
0answers
18 views

Should the weights be rotated when using SciPy full convolution?

I use SciPy's single.convolve2d in "full" mode to compute gradient w.r.t to convolution layer inputs. In my current implementation, I don't rotate filters as suggested by this article because I assume ...
1
vote
0answers
22 views

Neural-Networks - preferred method for training, classification v.s. regression

As a conclusion of their paper "Efficient Backprop" (http://yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf) (§10 Discussion and Conclusion), LeCun and others conlude that the preferred method for ...
1
vote
0answers
39 views

How can I regularize the output of a layer from scratch (without using Keras)?

I am trying to build a Convolutional Neural Network after reading notes from Stanford's cs231n course. I use ELU activation as activation function, and SoftMax as my classifier. Architecture is simple:...
1
vote
0answers
18 views

What is the difference between these 2 training scenarios?

I have a very large dataset and due to computational constraints, I have to divide the data into 20 parts (each part is around 1.5GB). I constructed a deep CNN model using Keras for this dataset. The ...
1
vote
0answers
26 views

Purpose of gamma multiplier in gradient boosting

looking through the mathematics of gradient boosting on the relevant wikipedia page, intuitively what is the purpose of the multiplier $\gamma_i$? This term does not appear in the following ...
1
vote
0answers
172 views

SKLearn Boston dataset gradient descent not working

I am trying to compare some simple methods for linear regression as an exercise. I have already used LinearRegression from the SKLearn library in python as well as the formula of linear regression. ...
1
vote
0answers
91 views

Gradient boosting, where did the constant go?

In the very early papers on gradient boosting, the ensemble would include a constant and a sum of base learners i.e. $F(X) = a_0 + \sum\limits_{i} a_i f_i(X)$ The constant is fitted first (i.e. if ...
1
vote
1answer
31 views

Pytorch - Gradient distribution between functions

https://colab.research.google.com/github/pytorch/tutorials/blob/gh-pages/_downloads/neural_networks_tutorial.ipynb Hi I am trying to understand the NN with pytorch. I have doubts in gradient ...
1
vote
0answers
23 views

Why does NAG cause unstable validation loss?

I'm building a neural network for a classification problem. When playing around with some hyperparameters, I was surprised to see that using Nesterov's Accelerated Gradient instead of vanilla SGD ...
1
vote
0answers
101 views

Updating Weight Using Updates on Related Data

Suppose $$ x=Ay $$ The $x$ is $M\times 1$, $y$ is $N \times 1$ and $A$ is $M\times N$ We have the data $x$ and would like to know what $y$ is. However, the matrix $A$ is too large for pseudo-...
1
vote
0answers
81 views

CartPole v1 - Simple backprop with 1 hidden layer

I'm trying to solve the CartPole-v1 problem from OpenAI by using backprop on a one-layer neural network - while updating the model at every time step using State action values (Q(s,a)). I'm unable to ...