265 views

### gradient descent in n dimensions

Gradient descent in $n$ dimensions. I'm learning about the downward gradient and the youtube videos and books only show a 2d curve as the slope drops to the minimum of the curve. My question is, ...
100 views

### Why does only Adam optimiser perform well in my case?

I am trying to classify texts into categories (one text can have multiple categories), to do this I used one-hot encoded labels (with ...
468 views

### Gradient descent in a noisy environment

How to know the right direction in a noisy environment? In the typical example of neural network learning, we can see several local minima. The gradient descent is choosing one local minimum and ...
10k views

### Why my network needs so many epochs to learn?

I'm working on a relation classification task for natural language processing and I have some questions about the learning process. I implemented a convolutional neural network using PyTorch, and I'm ...
451 views

I'm a double major in Math and CS interested in Machine Learning. I'm currently taking the popular Coursera course by Prof. Andrew. He's talking and explaining Gradient Descent but I can't avoid ...
62 views

### How to find learning rate decay?

Given the number of epochs, batch size and learning rate, is there a formula by which I can calculate the learning rate decay in mini batch SGD?
45 views

### Minimization algorithm that can consider gradient close to solution

I want to minimize a function which has sharp gradients close to each local minimum. Due to process tolerances, I want to find solutions which meet some minimum criterion (e.g. lower than x), but have ...
859 views

### Gradient Descent Python Implementation isnt converging

I'm trying to implement gradient descent in Python and following Andrew Ng course in order to follow the math. However, my implementation isn't working as I expected. It would be great if the ...
5k views

### Log loss vs accuracy for deciding between different learning rates?

While model tuning using cross validation and grid search I was plotting the graph of different learning rate against log loss and accuracy separately. Log loss When I used log loss as score in ...
132 views

### Why Gradient methods work in finding the parameters in Neural Networks?

After reading quite a lot of papers (20-30 or so), I feel that I am quite not understanding things. Let us focus on the supervised learnings (for example). Given a set of data \$\mathcal{D}_{train}=\{...
6k views

### What are the cases where it is fine to initialize all weights to zero

I've taken a few online courses in machine learning, and in general, the advice has been to choose random weights for a neural network to ensure that your neurons don't all learn the same thing, ...
1k views

### The connection between optimization and generalization

Optimization algorithms such as gradient descent or particle swarm can find a minima in a function. On the other hand, learning methods such as back-prop define learning as an optimization problem ...
11k views

### How to plot cost versus number of iterations in scikit learn?

One of the recommendations in the Coursera Machine Learning course when working with gradient descent based algorithms is: Debugging gradient descent. Make a plot with number of iterations on the x-...