Questions tagged [sgd]
Stochastic Gradient Descent (SGD) is an iterative algorithms used for objective function optimization used in machine learning models.
20 questions
1
vote
1
answer
62
views
Does using different optimizer change the loss landscape
I plot the landscape using this code, and I notice the landscape shape has changed a lot. My understanding is that the optimizer does not change the loss landscape. But now I'm confused if its just ...
- 127
1
vote
0
answers
23
views
How to calculate gradient of MSE in backpropagation? [duplicate]
I want to implement a neural network from scratch to solve linear regression by using backpropagation. I don't understand how to compute the gradient of the MSE cost function with respect to each ...
- 152
1
vote
1
answer
610
views
Difference between sklearn's LogisticRegression and SGDClassifier?
What is the difference between sklearn's LogisticRegression classifier and its SGDClassifier? I understand that the SGD is an ...
0
votes
1
answer
276
views
Gradient descent vs stochastic gradient descent vs mini-batch gradient descent with respect to working step/example
I am trying to understand the working of gradient descent, stochastic gradient descent and mini-batch gradient descent.
In case of gradient descent, gradient is computed on the entire dataset at each ...
- 217
1
vote
1
answer
1k
views
Understanding SGD for Binary Cross-Entropy loss
I'm trying to describe mathematically how stochastic gradient descent could be used to minimize the binary cross entropy loss.
The typical description of SGD is that I can find online is: $\theta = \...
- 13
1
vote
1
answer
177
views
How variable alpha changes SGDRegressor behavior for outlier?
I am using SGDRegressor with a constant learning rate and default loss function. I am curious to know how changing the alpha parameter in the function from 0.0001 to 100 will change regressor behavior....
- 111
1
vote
1
answer
30
views
Understanding Learning Rate in depth
I am trying to understand why the learning rate does not work universally. I have two different data sets and have tested out three learning rates 0.001 ,0.01 and 0.1 . For the first data set, I was ...
- 11
1
vote
2
answers
35
views
Multiple models have extreme differences during evaluation
My dataset has about 100k entries, 6 features, and the label is simple binary classification (about 65% zeros, 35% ones).
When I train my dataset on different models: random forest, decision tree, ...
- 13
1
vote
0
answers
61
views
How exactly do you implement SGD with momentum?
I am looking up sources to implement SGD with momentum, but they are giving me different equations.
(beta is the momentum hyper-parameter, ...
- 111
0
votes
2
answers
963
views
1
vote
2
answers
409
views
Why does using Gradient descent over Stochatic gradient descent improve performance?
Currently, I'm running two types of logistic regression.
logistic regression with SGD
logistic regression with GD
implemented as follows
...
- 479
0
votes
1
answer
2k
views
Learning rate of 0 still changes weights in Keras
I just trained a model (SGD) with keras and was wondering why the change of accuracy and loss from epoch to epoch doesn't really decrease that much when I lower the learning rate. So I tested what ...
- 80
6
votes
1
answer
4k
views
Changing the batch size during training
The choice of batch size is in some sense the measure of stochasticity :
On one hand, smaller batch sizes make the gradient descent more stochastic, the SGD can deviate significantly from the exact ...
1
vote
1
answer
2k
views
input shape of keras Sequential model
i am new to neural networks using keras,
i have the following train samples input shape (150528, 1235) and output shape is (154457, 1235) where 1235 is the training examples,
how to put the input ...
- 23
0
votes
0
answers
166
views
Problem of multi class classification (Sklearn TfidfVectorizer and SGDClassifier)
I do the (text) topic classification using TfidfVectorizer and SGDClassifier, literally I want to classify the website into categories (like Sport, Business etc).
Now, the problem is, that each ...
- 133
1
vote
2
answers
6k
views
Confused between optimizer and loss function
I always thought the SGD was a loss function then I read this on a notebook
...
3
votes
2
answers
637
views
The central idea behind SGD
Pr. Hinton in his popular course on Coursera refers to the following fact:
Rprop doesn’t really work when we have very large datasets and need to perform mini-batch weights updates. Why it doesn’t ...
- 14.2k
1
vote
3
answers
3k
views
Why Mini batch gradient descent is faster than gradient descent?
As I understand them:
Mini Batch Gradient Descent :
It takes a specified batch number say 32.
Evaluate loss on 32 examples.
Update weights.
Repeat until every example is complete.
Repeat till a ...
- 709
4
votes
2
answers
332
views
How is Stochastic Gradient Descent(SGD) used like Mini Batch Gradient Descent(MBGD)?
As I know, Gradient Descent(GD) has three variants which are:
1- Batch Gradient Descent(BGD): processes all the training examples for each iteration of gradient descent.
2- Stochastic Gradient Descent(...
- 1,177
3
votes
2
answers
481
views
Explanations about ADAM Optimizer algorithm
I'm a beginner in Machine learning and I'm searching for some optimizer for the gradient descent. I've searched many topics about that, and did a state of art of all these optimizers. I have just one ...
