1
$\begingroup$

Currently, I'm running two types of logistic regression.

  1. logistic regression with SGD
  2. logistic regression with GD

implemented as follows

SGD= SGDClassifier(loss="log",max_iter=1000,penalty='l1',alpha=0.001)
logreg = LogisticRegression(solver='liblinear', max_iter=100, penalty='l1', C=0.1)

nevermind the hyperparameters as I've used GridsearchCV and tried multiple combinations.

When calculating accuracy logistic with GD performs better than SGD. I want to understand why this is the case, is using GD instead SGD one way to mitigate underfitting model?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
0
$\begingroup$

SGD has a regularization effect and finds the solution faster. GD on the other hand takes a look at whole data and finds the next best step.

SGD may be come to optimal global minima but GD can. But GD is not practical with large data.

Improve this answer
$\endgroup$
2
  • $\begingroup$ I want to understand why SGD's ability to reach solution faster impact performance where loss function is convex? $\endgroup$
    – haneulkim
    Commented May 12, 2021 at 9:52
  • $\begingroup$ Being convex doesn't mean the solution will be reached because the data can have interdependency which will lead to many minimas but not all will be optimum i.e. global. $\endgroup$
    – Abhishek Verma
    Commented May 12, 2021 at 12:00
0
$\begingroup$

Gradient Descent should have better results as it runs in your whole data. Stochastic Gradient Descent looks at batches, making it useful for big data. Batches( or subset) make it run faster but, it can get converge at a local minimum.

On Wikipedia you can find the following citation :

SGD replaces the actual gradient (calculated from the entire data set) by an estimate thereof (calculated from a randomly selected subset of the data). Especially in high-dimensional optimization problems, this reduces the computational burden, achieving faster iterations in trade for a lower convergence rate

SGD vs GD

Improve this answer
$\endgroup$
2
  • $\begingroup$ SGD randomly selects one data, using subset is called mini batch GD and using full data is GD. In conext of logistic regression where it uses log loss ( in case of other loss im not sure) which is a convex function therefore it would converge eventually b.c. there is only one global minimum. $\endgroup$
    – haneulkim
    Commented May 12, 2021 at 9:51
  • $\begingroup$ I guess the difference bt one sample and mini batch GD is the N parameter. How do you know that it only has one minimum? as @Abhishek pointed out that is not the case $\endgroup$
    – Carlos Mougan
    Commented May 12, 2021 at 12:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.