# Any way to implement sequential incremental stochastic gradient descent algorithm in logistic regression?

I am trying to program incremental stochastic gradient descent (ISGD) algorithm in logistic regression. Initially, I coded respective logistic regression' loss function and its gradient, also got some idea to proceed rest of workflow. But, I have no idea how to apply sequential operation in incremental stochastic gradient descent algorithm which can be used in the respective logistic regression. How can I implement the sequential operation in incremental SGD? Any way to make this happen in Python? How can I do that? Any idea?

Initial implementation

import numpy as np
import scipy as sp
import sklearn as sl
from scipy import special as ss
from  sklearn import datasets

## load input dataset
X_train, y_train=datasets.load_svmlight_file('/path/to/train_dataset')
X_test,y_test=datasets.load_svmlight_file('/path/to/train_dataset.txt',
n_features=X_train.shape[1])

# logistic loss function
def lossFunc(x_i,y_i,w):
w.resize((w.shape[0],1))
y_i.resize((y_i.shape[0],1))

lossFnc=ss.log1p(1+np.nan_to_num(ss.expm1(-y_i* np.dot(x_i,w,))))
rslt=np.float(lossFnc)
return rslt

#gradient function

def gradFnc(X,y,w):
w.resize((w.shape[0],1))
y.resize((y.shape[0],1))

gradF1=-y*np.nan_to_num(ss.expm1(-y))
gradF2=gradF1/(1+np.nan_to_num(ss.expm1(-y*np.dot(X,w))))
gradF3=gradF2.resize(gradF2.shape[0],)
return gradF3

class ISGD:
def _init_(self, learnRate=0.0001, num_iter=100, verbose=False):
self.w=None
self.learnRate=learnRate
self.verbose=verbose
self.num_iter=num_iter

def fitt(self, X,y):
n,d=X.shape
self.w=np.zeros(shape=(d,))

for i in range(self.num_iter):
print ("\n:", "Iteration:", i)

grd=gradFnc(self.w, X,y)
grd.resize((grd.shape[0],1))
self.w=self.w-grd
print "Loss:", lossFunc(self.w,X,y)

return self

def predict(x_i, w):
y_hat=w[0]
for idx in range(len(x_i)-1):
y_hat+=w[i+1]*x_i[idx]
return 1.0/(1.0+np.nan_to_num(ss.expm1(-y_hat)))

def update_weights(x_i, y_i,w):
lr=0.8
yhat=predict(x_i, w)
error=y_i-yhat
return w+lr*(y_i-yhat)*x_i


How to proceed rest of workflow?

Here is blog about HogWild! for parallel machine learning. The particular interpretation of incremental SGD can be found here: hogwild! algorithm for logistic regression.

Now I have no idea how to apply sequential operation in incremental SGD which can be used in the respective logistic regression. How can I make this happen? Is there any efficient workaround to implement sequential incremental SGD algorithm for logistic regression? What is the efficient programming pipeline to accomplish the task that I stated above? Any more thoughts?