Linked Questions

0
votes
2answers
265 views

gradient descent in n dimensions

Gradient descent in $n$ dimensions. I'm learning about the downward gradient and the youtube videos and books only show a 2d curve as the slope drops to the minimum of the curve. My question is, ...
1
vote
3answers
100 views

Why does only Adam optimiser perform well in my case?

I am trying to classify texts into categories (one text can have multiple categories), to do this I used one-hot encoded labels (with ...
1
vote
3answers
468 views

Gradient descent in a noisy environment

How to know the right direction in a noisy environment? In the typical example of neural network learning, we can see several local minima. The gradient descent is choosing one local minimum and ...
12
votes
3answers
10k views

Why my network needs so many epochs to learn?

I'm working on a relation classification task for natural language processing and I have some questions about the learning process. I implemented a convolutional neural network using PyTorch, and I'm ...
1
vote
3answers
451 views

Gradient Descent Convergence

I'm a double major in Math and CS interested in Machine Learning. I'm currently taking the popular Coursera course by Prof. Andrew. He's talking and explaining Gradient Descent but I can't avoid ...
0
votes
1answer
62 views

How to find learning rate decay?

Given the number of epochs, batch size and learning rate, is there a formula by which I can calculate the learning rate decay in mini batch SGD?
0
votes
2answers
45 views

Minimization algorithm that can consider gradient close to solution

I want to minimize a function which has sharp gradients close to each local minimum. Due to process tolerances, I want to find solutions which meet some minimum criterion (e.g. lower than x), but have ...
0
votes
2answers
859 views

Gradient Descent Python Implementation isnt converging

I'm trying to implement gradient descent in Python and following Andrew Ng course in order to follow the math. However, my implementation isn't working as I expected. It would be great if the ...
7
votes
4answers
5k views

Log loss vs accuracy for deciding between different learning rates?

While model tuning using cross validation and grid search I was plotting the graph of different learning rate against log loss and accuracy separately. Log loss When I used log loss as score in ...
4
votes
2answers
132 views

Why Gradient methods work in finding the parameters in Neural Networks?

After reading quite a lot of papers (20-30 or so), I feel that I am quite not understanding things. Let us focus on the supervised learnings (for example). Given a set of data $\mathcal{D}_{train}=\{...
7
votes
2answers
6k views

What are the cases where it is fine to initialize all weights to zero

I've taken a few online courses in machine learning, and in general, the advice has been to choose random weights for a neural network to ensure that your neurons don't all learn the same thing, ...
5
votes
1answer
1k views

The connection between optimization and generalization

Optimization algorithms such as gradient descent or particle swarm can find a minima in a function. On the other hand, learning methods such as back-prop define learning as an optimization problem ...
8
votes
2answers
11k views

How to plot cost versus number of iterations in scikit learn?

One of the recommendations in the Coursera Machine Learning course when working with gradient descent based algorithms is: Debugging gradient descent. Make a plot with number of iterations on the x-...
2
votes
1answer
195 views

Optimization methods used in machine learning

I don't have too much knowledge in the field of ML, but from my naive point of view it always seems that some variant of gradient descent is used when training neutral networks. As such, I was ...
1
vote
2answers
658 views

Meaning of Perceptron optimal weights

Im studying the perceptron algorithm. I know that we can use the weights w as the coefficients of the hyperplane that will separate the vectors that need a classifications In every web page i read ...

15 30 50 per page