# Why does logistic regression in Spark and R return different models for the same data?

I've compared the logistic regression models on R (glm) and on Spark (LogisticRegressionWithLBFGS) on a dataset of 390 obs. of 14 variables.

The results are completely different in the intercept and the weights. How to explain this?

Here is the results of Spark (LogisticRegressionWithLBFGS) :

model.intercept  :
1.119830027739959
model.weights :
GEST    0.30798496002530473
DILATE  0.28121771009716895
EFFACE  0.01780105068588628
CONSIS -0.22782058111362183
CONTR  -0.8094592237248102
MEMBRAN-1.788173534959893
AGE    -0.05285751197750732
STRAT  -1.6650305527536942
GRAVID  0.38324952943210994
PARIT  -0.9463956993328745
DIAB   0.18151162744507293
TRANSF -0.7413500749909346
GEMEL  1.5953124037323745


Here is the result of R :

             Estimate Std. Error z value Pr(>|z|)
(Intercept)  3.0682091  3.3944407   0.904 0.366052
GEST         0.0086545  0.1494487   0.058 0.953821
DILATE       0.4898586  0.2049361   2.390 0.016835 *
EFFACE       0.0131834  0.0059331   2.222 0.026283 *
CONSIS       0.1598426  0.2332670   0.685 0.493196
CONTR        0.0008504  0.5788959   0.001 0.998828
MEMBRAN     -1.5497870  0.4215416  -3.676 0.000236 ***
AGE         -0.0420145  0.0326184  -1.288 0.197725
STRAT       -0.3781365  0.5860476  -0.645 0.518777
GRAVID       0.1866430  0.1522925   1.226 0.220366
PARIT       -0.6493312  0.2357530  -2.754 0.005882 **
DIAB         0.0335458  0.2163165   0.155 0.876760
TRANSF      -0.6239330  0.3396592  -1.837 0.066219 .
GEMEL        2.2767331  1.0995245   2.071 0.038391 *
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

• I wouldn't call Limited-memory BFGS a "simple logistic regression". Except, I guess, in the sense that modern libraries make complex techniques very accessible :) – nealmcb May 7 '15 at 17:08
• You are right. Do you know a way to implement Limited-memory BFGS with R ? – SparkUser May 11 '15 at 8:15
• Also, suspect if it has something to do with the loss function csie.ntu.edu.tw/~cjlin/papers/spark-liblinear/… (Sec - IIIA) @SparkUser - is it possible to run LogisticRegressionWithLBFGS without the defaults and then compare the coefficients with the default R - glm. – user45409 Apr 23 '16 at 17:47

A quick glance at the docs for LogisticRegressionWithLBFGS indicates that it uses feature scaling and L2-Regularization by default. I suspect that R's glm is returning a maximum likelihood estimate of the model while Spark's LogisticRegressionWithLBFGS is returning a regularized model estimate. Note how the estimated model weights of the Spark model are all smaller in magnitude than those in the R model.
I'm not sure whether or not glm in R is implementing feature scaling, but this would also contribute to different model values.
• Here, check the documentation of function glmnet from glmnet package and look and parameter standardize http://cran.r-project.org/web/packages/glmnet/glmnet.pdf - there also is possibleto use regularization – Marcin Kosiński Jun 1 '15 at 10:03