Questions tagged [sgd]

Stochastic Gradient Descent (SGD) is an iterative algorithms used for objective function optimization used in machine learning models.

Filter by
Sorted by
Tagged with
1 vote
1 answer
26 views

Understanding SGD for Binary Cross-Entropy loss

I'm trying to describe mathematically how stochastic gradient descent could be used to minimize the binary cross entropy loss. The typical description of SGD is that I can find online is: $\theta = \...
user avatar
  • 13
0 votes
0 answers
20 views

ResNet: Derive the gradient matrices w.r.t. W1 and W2 and backprop equation in a Residual Network

How would I go about step by step deriving stochastic gradient matrices w.r.t. W1 and W2 and backpropagation equation in a residual block that is a part of a larger ResNet network with forward ...
user avatar
  • 1
1 vote
1 answer
18 views

How variable alpha changes SGDRegressor behavior for outlier?

I am using SGDRegressor with a constant learning rate and default loss function. I am curious to know how changing the alpha parameter in the function from 0.0001 to 100 will change regressor behavior....
user avatar
  • 111
1 vote
0 answers
14 views

Understanding Learning Rate in depth

I am trying to understand why the learning rate does not work universally. I have two different data sets and have tested out three learning rates 0.001 ,0.01 and 0.1 . For the first data set, I was ...
user avatar
  • 11
0 votes
0 answers
12 views

How to compute constant c for PCA features before SGDClassifier as advised in Scikit documentation?

In the documentation for SGDClassifier here, it is stated; If you apply SGD to features extracted using PCA we found that it is often wise to scale the feature values by some constant c such that the ...
user avatar
  • 111
0 votes
0 answers
13 views

Understanding the step of SGD for binary classification

I cannot understand the step of SGD for binary classification. For example, we have $y$ - true labels $\in \{0,1\}$ and $p=f_\theta(x)$-predicted labels $\in [0,1]$. Then, the update step of SGD is ...
user avatar
  • 101
0 votes
0 answers
18 views

Is learning_rate linear with the time to converge using AdamOpt?

Say that both learning rates 1e-3,1e-4 leading to the same solution (not too high or too small). In terms of convergence by the amount of epochs, does ...
user avatar
1 vote
2 answers
24 views

Multiple models have extreme differences during evaluation

My dataset has about 100k entries, 6 features, and the label is simple binary classification (about 65% zeros, 35% ones). When I train my dataset on different models: random forest, decision tree, ...
user avatar
  • 13
1 vote
0 answers
16 views

How exactly do you implement SGD with momentum?

I am looking up sources to implement SGD with momentum, but they are giving me different equations. (beta is the momentum hyper-parameter, ...
user avatar
  • 111
0 votes
2 answers
216 views

Can't use The SGD optimizer

I am using the following code: ...
user avatar
  • 7
0 votes
0 answers
15 views

Estimating a rbf kernel SVM, followed by Stochastic Gradient Descent

I wanna estimate a rbf SVM to predict property prices. My data set has 11 features and roughly 57,000 rows. When I set C=10, R^2 is about 0.88 while MSE and RMSE are 0.1191 and 0.3451. The results are ...
user avatar
0 votes
0 answers
42 views

Stochastic Gradient Region of Confusion

I have come across the following diagram which explains the behavior of SGD graphically. Based on this graphical representation, the gradient of the individual data tend to fluctuate more when it ...
user avatar
1 vote
2 answers
55 views

Why does using Gradient descent over Stochatic gradient descent improve performance?

Currently, I'm running two types of logistic regression. logistic regression with SGD logistic regression with GD implemented as follows ...
user avatar
  • 343
0 votes
1 answer
177 views

Learning rate of 0 still changes weights in Keras

I just trained a model (SGD) with keras and was wondering why the change of accuracy and loss from epoch to epoch doesn't really decrease that much when I lower the learning rate. So I tested what ...
user avatar
  • 70
6 votes
1 answer
913 views

Changing the batch size during training

The choice of batch size is in some sense the measure of stochasticity : On one hand, smaller batch sizes make the gradient descent more stochastic, the SGD can deviate significantly from the exact ...
user avatar
1 vote
1 answer
2k views

input shape of keras Sequential model

i am new to neural networks using keras, i have the following train samples input shape (150528, 1235) and output shape is (154457, 1235) where 1235 is the training examples, how to put the input ...
user avatar
  • 23
0 votes
0 answers
104 views

Problem of multi class classification (Sklearn TfidfVectorizer and SGDClassifier)

I do the (text) topic classification using TfidfVectorizer and SGDClassifier, literally I want to classify the website into categories (like Sport, Business etc). Now, the problem is, that each ...
user avatar
  • 133
1 vote
2 answers
3k views

Confused between optimizer and loss function

I always thought the SGD was a loss function then I read this on a notebook ...
user avatar
2 votes
2 answers
460 views

The central idea behind SGD

Pr. Hinton in his popular course on Coursera refers to the following fact: Rprop doesn’t really work when we have very large datasets and need to perform mini-batch weights updates. Why it doesn’t ...
user avatar
  • 13.3k