I have a difficulty where to start the implementation of incremental stochastic gradient descent algorithm and its respective implementation in logistic regression. I don't quite understand this algorithm and there are few sources to explain it with crystal-clear interpretation and possible demo code. I am quite new with ML algorithm and I have no idea which would be efficient workaround to solve this problem.
In particular, the problem I am studying is to implement hogWild! algorithm for logistic regression, which asks me to program incremental SGD algorithm with a sequential order. Can anyone give me a general idea or possible pipeline to make this happen in python?
logistic loss function and gradient
Here is my implementation:
import numpy as np
import scipy as sp
import sklearn as sl
from scipy import special as ss
from sklearn import datasets
X_train, y_train=datasets.load_svmlight_file('/path/to/train_dataset')
X_test,y_test=datasets.load_svmlight_file('/path/to/train_dataset.txt',
n_features=X_train.shape[1])
class ISGD:
def lossFunc(X,y,w):
w.resize((w.shape[0],1))
y.resize((y.shape[0],1))
lossFnc=ss.log1p(1+np.nan_to_num(ss.expm1(-y* np.dot(X,w,))))
rslt=np.float(lossFnc)
return rslt
def gradFnc(X,y,w):
w.resize((w.shape[0],1))
y.resize((y.shape[0],1))
gradF1=-y*np.nan_to_num(ss.expm1(-y))
gradF2=gradF1/(1+np.nan_to_num(ss.expm1(-y*np.dot(X,w))))
gradF3=gradF2.resize(gradF2.shape[0],)
return gradF3
def _init_(self, learnRate=0.0001, num_iter=100, verbose=False):
self.w=None
self.learnRate=learnRate
self.verbose=verbose
self.num_iter=num_iter
def fitt(self, X,y):
n,d=X.shape
self.w=np.zeros(shape=(d,))
for i in range(self.num_iter):
print ("\n:", "Iteration:", i)
grd=gradFnc(self.w, X,y)
grd.resize((grd.shape[0],1))
self.w=self.w-grd
print "Loss:", lossFunc(self.w,X,y)
return self
Seems my above implementation has some problems. Can anyone help me how to correct that? Plus, I don't have a solid idea how to implement incremental SGD sequentially. How can I make this happen? Any idea?