Linked Questions

20
votes
2answers
7k views

local minima vs saddle points in deep learning

I heard Andrew Ng (in a video I unfortunately can't find anymore) talk about how the understanding of local minima in deep learning problems has changed in the sense that they are now regarded as less ...
8
votes
2answers
10k views

How to plot cost versus number of iterations in scikit learn?

One of the recommendations in the Coursera Machine Learning course when working with gradient descent based algorithms is: Debugging gradient descent. Make a plot with number of iterations on the x-...
7
votes
2answers
5k views

What are the cases where it is fine to initialize all weights to zero

I've taken a few online courses in machine learning, and in general, the advice has been to choose random weights for a neural network to ensure that your neurons don't all learn the same thing, ...
11
votes
3answers
6k views

Why my network needs so many epochs to learn?

I'm working on a relation classification task for natural language processing and I have some questions about the learning process. I implemented a convolutional neural network using PyTorch, and I'm ...
6
votes
2answers
5k views

Does MLP always find local minimum

In linear regression we use the following cost function which is a convex function: We Use the following cost function in ...
7
votes
4answers
4k views

Log loss vs accuracy for deciding between different learning rates?

While model tuning using cross validation and grid search I was plotting the graph of different learning rate against log loss and accuracy separately. Log loss When I used log loss as score in ...
5
votes
1answer
876 views

The connection between optimization and generalization

Optimization algorithms such as gradient descent or particle swarm can find a minima in a function. On the other hand, learning methods such as back-prop define learning as an optimization problem ...
0
votes
2answers
801 views

Gradient Descent Python Implementation isnt converging

I'm trying to implement gradient descent in Python and following Andrew Ng course in order to follow the math. However, my implementation isn't working as I expected. It would be great if the ...
1
vote
3answers
376 views

Gradient Descent Convergence

I'm a double major in Math and CS interested in Machine Learning. I'm currently taking the popular Coursera course by Prof. Andrew. He's talking and explaining Gradient Descent but I can't avoid ...
1
vote
2answers
578 views

Meaning of Perceptron optimal weights

Im studying the perceptron algorithm. I know that we can use the weights w as the coefficients of the hyperplane that will separate the vectors that need a classifications In every web page i read ...
1
vote
3answers
370 views

Gradient descent in a noisy environment

How to know the right direction in a noisy environment? In the typical example of neural network learning, we can see several local minima. The gradient descent is choosing one local minimum and ...
4
votes
2answers
121 views

Why Gradient methods work in finding the parameters in Neural Networks?

After reading quite a lot of papers (20-30 or so), I feel that I am quite not understanding things. Let us focus on the supervised learnings (for example). Given a set of data $\mathcal{D}_{train}=\{...
1
vote
1answer
184 views

Optimization methods used in machine learning

I don't have too much knowledge in the field of ML, but from my naive point of view it always seems that some variant of gradient descent is used when training neutral networks. As such, I was ...
0
votes
2answers
157 views

gradient descent in n dimensions

Gradient descent in $n$ dimensions. I'm learning about the downward gradient and the youtube videos and books only show a 2d curve as the slope drops to the minimum of the curve. My question is, ...
1
vote
3answers
93 views

Why does only Adam optimiser perform well in my case?

I am trying to classify texts into categories (one text can have multiple categories), to do this I used one-hot encoded labels (with ...

15 30 50 per page