Gradient Descent is an algorithm for finding the minimum of a function. It iteratively calculates partial derivatives (gradients) of the function and descends in steps proportional to those partial derivatives. One major application of Gradient Descent is fitting a parameterized model to a set of data: the function to be minimized is an error function for the model.

109 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
472 views

differences between LSQR and FTRL when working with very sparse data

I have a 2M instances dataset with millions of very very sparse dummy variables created using the hashing trick = ...
79 views

Learning parameters when loss is a piecewise function

I have a network to generate a single number $T$. I know in advance: a property of the loss function is that, when $T \in [a_1, a_2]$, the loss has the same value $L_1$; when $T \in [a_2, a_3]$, the ...
70 views

Why the sigmoid activation function results in sub-optimal gradient descent?

I need some help understanding the second shortcoming of the sigmoid activation function as described in this video from Stanford. She says that because the output of sigmoid is always positive, that ...
248 views

Plotting Gradient Descent in 3d - Contour Plots

I have generated 3 parameters along with the cost function. I have the $\theta$ lists and the cost list of 100 values from the 100 iterations. I would like to plot the last 2 parameters against cost ...
251 views

Fast Python implementation of the gradient descent

I'm looking for fast Python implémentations of gradient descent optimization algorithm. I have a convex problem , with no constraint, so for now I'm using the BFGS algorithm implemented in scikit-...
22 views

Issues with self-implemented logistic regression

I am trying to self-implement a logistic regression algorithm to do some self-learning but I am having a bit of trouble with achieving similar accuracy to the logistic regression of sklearn. Here is ...
183 views

The update equation of AdaGrad is as follows: I understand that sparse features have small updates and this is a problem. I understand that the idea of AdaGrad is to make the update speed (learning ...
45 views

Adam's optimization has the fololwing parameter update rule : $$\theta_{t+1} = \theta_{t} - \alpha*\dfrac{m_t}{\sqrt{v_t + \epsilon}}$$ where $$m_t \text{ is first moment of gradients and} \space ... 1answer 74 views Row-wise Jacobian with pytorch Suppose I have f:\mathbb{R}^{d_i}\to\mathbb{R}^{d_o}. Let X \in \mathbb{R}^{n \times d_i} and I apply f to each row of X, obtaining Y = f(X) \in \mathbb{R}^{n \times d_o}. I would like to ... 0answers 48 views Maximum Entropy Policy Gradient Derivation I am reading through the paper on Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review by Sergey Levine. I am having a difficulty in understanding this part of the ... 0answers 61 views How to determine the convergence of Stochastic Gradient Descent? While coding the batch gradient descent, it is easy to code the convergence as after each iterations the cost moves towards minimum and when the change in cost tends to approach a pre-defined number, ... 1answer 388 views Intractability in Variational Autoencoders I'm having difficulty understanding when integrals are intractable in variational inference problems. In a variational autoencoder with observation x and latent variable z we want to maximize data ... 0answers 27 views Get derivatives from your NN How can I get the gradient of a node in the NN with respect to another one? I need to train a NN, which for the sake of simplicity has 2 neurons as input (x, y), a neuron as a bottleneck (z), and 2 ... 1answer 819 views multilayer perceptron do not converge I have been coding my own multi layer perceptron in MATLAB and it can be compiled without error. My training data features,x, has values from 1 to 360, and training data output, y, has the value of ... 0answers 181 views Explanations about ADAM Optimizer algorithm I'm a beginner in Machine learning and i'm searching for some optimizer for the gradient descent. I've searched many topics about that, and did a state of art of all these optimizers. I have just one ... 0answers 156 views Vowpal Wabbit Online Normalization — Possible to parallelize? Vowpal Wabbit (VW) uses online normalization as explained here [1]. When running VW with multiple workers, workers synchronize their models with an AllReduce at the end of each epoch. Is it possible ... 0answers 146 views SGD learning gets stuck when using a max pooling layer (but it works fine with just conv + fc) I'm working on a CNN library for a university project and I'm having some trouble implementing the backpropagation through the max pooling layer. Please note that the whole thing was built from ... 0answers 95 views What are some machine learning problems that can be attacked with continuous multiobjective optimization? I am working on continuous vector optimization, and hence continuous multiobjective optimization is a particular case. I am interested in finding applications in machine learning for this problems. Is ... 0answers 124 views Why is each successive tree in GBM fit on the negative gradient of the loss function? Page 359 of Elements Of Statistical Learning 2nd edition says the below. Can someone explain the intuition & simplify it in layman terms? Questions What is the reason/intuition & math ... 0answers 203 views Non-linear data preprocessing before mini-batch gradient descent I found some realisation of pretty interesting ML algorithm. https://github.com/EderSantana/DeepEEG We have dataset X, matrix of spacial filter W, matrix of time filter V, and matrices U and B - ... 0answers 76 views Regression problem - too complex for gradient descent I try to predict temperatures values as function of time and different parameters. The temperature curve look like a "ramp" with some "gauss peaks" on regular intervals. So, I try to build a ... 0answers 285 views How do I deal with non-IID data in gradient boosted random forest (for stock market)? I am working on a stock market decision system. I have currently centered on gradient boosting as the likely best machine learning solution for the problem. However, I have 2 fundamental issues with ... 0answers 29 views SVM with gradient descent The constrained optimization problem in SVM is given by min 1/2 ||w||^2 s.t y(i)(w^T x(i) + b >= 1 for all i Now converting this to an unconstrained optimization problem gives the lagriangian L as ... 0answers 41 views How are batch gradients computed on embedding layers? Consider the following model, which is more or less a 12-dimensional vector lookup table with 10 rows, initialized to all zeros. ... 0answers 23 views Backpropagation Mathematics with Sigmoid Output Activation and Cross Entropy Loss I am deriving a Weight update for a simple toy network with a Sigmoid Output Layer. I need some help double checking my math to make sure I did it correctly. I am using Cross-Entropy Loss as my Loss ... 0answers 24 views Why is the exploding/vanishing gradient problem not solved by line search? The problem of vanishing gradients is basically that since our step size is proportional to the gradient, if the gradient is very small, it might take a long time to reach a local minimum. So why don'... 1answer 42 views Why do we only care about convex functions when doing Gradient Descent/SGD? I mean I know why we specifically care about convex functions: it's because their local minimum are also global, and so you just have to "follow a path which goes down" to find the minima of ... 1answer 68 views Do batch GD and stochastic GD give the same results? If a neural network is trained on a dataset of M samples for N epochs, do batch GD and SGD give the same result? Is SGD is faster because utilize the hardware better? I am asking because I figured out ... 1answer 51 views How to find slope of curve at certain points how to find slope at certain points circled in blue in below curve ? Are these below 2 approaches valid ? though they give different results . How to automatically find the points where the slope ... 1answer 57 views Why does Siamese neural networks use tied weights and how do they work? Reading this paper on one-shot learning "Siamese Neural Networks for One-shot Image Recognition" I was introduced to the idea of Siamese Neural Networks. What I did not fully grasp was what they ... 0answers 56 views Derivative of Loss wrt bias term I read this and have an ambiguity. I try to understand well how to calculate the derivative of Loss w.r.t to bias. In this question, we have this definition: ... 0answers 18 views Tuning parameters for gradient boosting/xgboost In practice, which parameter do you typically tune first? Do you tune the learning rate (or step size) first? and then tune the total number of iterations? And how do you go about tuning these ... 0answers 15 views Matlab Optimization. Meaning of warning: “The slope should be 2. It appears to be 1.” I'm using the manopt package to solve some optimization problems in matlab. The problem is of the form. problem.cost = @(x) f(x) problem.egrad = @(x) g(x) After the problem definition, I check ... 0answers 15 views Gradient calculation for proportional odds (logistic) model I am trying to calculate a gradient for a proportional odds model. http://fa.bianp.net/blog/2013/logistic-ordinal-regression/ What steps are required to take the derivative with respect to w?$$\...
68 views

I have manually created a random data set around some mean value and I have tried to use gradient descent linear regression to predict this simple mean value. I have done exactly like in the manual ...
50 views

Transposed Convolution without using Python built-in functions

Amateur here: How can we write a 2D transposed convolution (aka deconvolution) using the steepest descent method given the following restrictions: cannot use any Python built-in functions cannot ...
188 views

grad-cam implementation on mobilenet SSD network

Below is a gradcam implementation for a standard image classifier : ...
24 views

Is there a gradient descent-based optimization algorithm that works with non-linear constraints?

I have a function to optimize with ca. 200 parameters + one constraint (sum of squares of the parameters must be equal one) This problem can be solved using Lagrange Multipliers and my intuition ...
45 views

Can we optimize heterogeneous parameters of RBF Network using Gradient Descent?

There're three parameters in the Radial Basis Function Networks (RBFN). Centers of RBFs Width of RBFs Weights of RBFs It's a fact that Weights can be easily updated using a simple Gradient Descent. ...
18 views

Should the weights be rotated when using SciPy full convolution?

I use SciPy's single.convolve2d in "full" mode to compute gradient w.r.t to convolution layer inputs. In my current implementation, I don't rotate filters as suggested by this article because I assume ...
22 views

Neural-Networks - preferred method for training, classification v.s. regression

As a conclusion of their paper "Efficient Backprop" (http://yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf) (§10 Discussion and Conclusion), LeCun and others conlude that the preferred method for ...
39 views

How can I regularize the output of a layer from scratch (without using Keras)?

I am trying to build a Convolutional Neural Network after reading notes from Stanford's cs231n course. I use ELU activation as activation function, and SoftMax as my classifier. Architecture is simple:...
18 views

What is the difference between these 2 training scenarios?

I have a very large dataset and due to computational constraints, I have to divide the data into 20 parts (each part is around 1.5GB). I constructed a deep CNN model using Keras for this dataset. The ...
26 views

Purpose of gamma multiplier in gradient boosting

looking through the mathematics of gradient boosting on the relevant wikipedia page, intuitively what is the purpose of the multiplier $\gamma_i$? This term does not appear in the following ...
172 views

SKLearn Boston dataset gradient descent not working

I am trying to compare some simple methods for linear regression as an exercise. I have already used LinearRegression from the SKLearn library in python as well as the formula of linear regression. ...
91 views

Gradient boosting, where did the constant go?

In the very early papers on gradient boosting, the ensemble would include a constant and a sum of base learners i.e. $F(X) = a_0 + \sum\limits_{i} a_i f_i(X)$ The constant is fitted first (i.e. if ...
31 views

23 views

Why does NAG cause unstable validation loss?

I'm building a neural network for a classification problem. When playing around with some hyperparameters, I was surprised to see that using Nesterov's Accelerated Gradient instead of vanilla SGD ...
Suppose $$x=Ay$$ The $x$ is $M\times 1$, $y$ is $N \times 1$ and $A$ is $M\times N$ We have the data $x$ and would like to know what $y$ is. However, the matrix $A$ is too large for pseudo-...